From 140d763f5252fe97dcc875f7e37a36521167bf86 Mon Sep 17 00:00:00 2001 From: David Ndungu Date: Sun, 23 Aug 2026 16:46:16 -0700 Subject: [PATCH 1/2] fix(kernels): never launch an unresolved optional kernel symbol Closes #180. The null pointer in the #180 crash is the FUNCTION pointer, not a buffer. In the reported trace `_Cfunc_ccall_wrapper(0x0, ...)`, the 0x0 is Ccall's `fn` argument; the input and output device pointers in the frame above are both non-null, so the "dst was never allocated" hypothesis in the issue is not what happened. launch_repeat_interleave_f32 is an OPTIONAL symbol (purego.go optionalSyms): when dlsym cannot find it, openKernelLib leaves the pointer at 0 and continues, on the documented contract that "callers check before use". RepeatInterleaveF32 checked only `k == nil`, so on any deployment whose libkernels.so predates the kernel it called cuda.Ccall(0, ...) -- a jump to address 0, i.e. SIGSEGV PC=0x0 inside cgo. That kills the process outright, which is why zerfoo's Reshape -> Repeat -> Reshape fallback never ran and why every GQA model died on GPU. The deployment in question is real and measured: the GB10 standing gate puts /opt/zerfoo/lib on LD_LIBRARY_PATH and loads a prebuilt libkernels.so from there rather than building this tree. That library exports launch_repeat but not launch_repeat_interleave_f32. RepeatInterleaveF32 was not a one-off. An AST gate over the package found 34 cuda.Ccall sites launching optional symbols with no zero-check, across 14 wrappers; 22 of those symbols are absent from the currently deployed library, so they are live process kills rather than latent ones. All 34 now check. Also adds IsRepeatInterleaveF32Supported so callers and tests can tell "the fused kernel ran" from "the engine silently fell back", which GPUEngine.RepeatInterleave otherwise hides. Tests (ADR 091): - TestRepeatInterleaveF32_UnresolvedSymbol_ReturnsError reproduces the exact #180 trace with no GPU required. Against the unfixed wrapper it does not fail, it CRASHES the test binary: SIGSEGV: segmentation violation PC=0x0 m=0 sigcode=1 addr=0x0 cuda._Cfunc_ccall_wrapper(0x0, ...) kernels.RepeatInterleaveF32(0x460686809eb8, 0x460686809db8, 1, 2, 2, 8, 2, 0) internal/cuda/kernels/fused_repeat_interleave_purego.go:23 It is paired with a no-library case that must produce a DISTINCT error, so a blanket "always return an error" cannot satisfy it. - TestOptionalKernelSymbolsAreGuarded is the class gate: it parses optionalSyms and walks every wrapper's AST, failing on any optional launch whose function pointer is not compared to 0 first. Red before this change with all 34 sites named; green after. It refuses to pass if its own extraction finds no symbols or no call sites. - compute.TestGPUEngine_RepeatInterleave_CPUParity is the engine-parity harness for the op. Because RepeatInterleave falls back on any error, it asserts the fused kernel is actually present first and FAILS rather than skips otherwise -- verified red against the stale /opt/zerfoo/lib library. Values are compared exactly (a pure gather does no arithmetic; a float tolerance here would repeat the #182 mistake) against a closed-form reference rather than a second ztensor call, and each element encodes its own source coordinate so a wrong head mapping cannot compare equal. A self-test asserts that encoding is discriminating before the comparison is trusted (docs/lore.md L-0009); mutating the reference mapping to q%numKV turns both the self-test and the comparison red. gradcheck (ADR 091 harness 1) is not extended here: it is defined over graph.Node implementations, and RepeatInterleave is an engine-level forward gather with no graph.Node and no analytic Backward in ztensor. --- compute/gpu_repeat_interleave_parity_test.go | 193 ++++++++++++ internal/cuda/kernels/counter_purego.go | 6 + .../cuda/kernels/elementwise_bf16_purego.go | 39 +++ .../cuda/kernels/elementwise_fp16_purego.go | 6 + internal/cuda/kernels/fp8_ops_purego.go | 12 + .../cuda/kernels/fused_adamw_bf16_purego.go | 3 + .../cuda/kernels/fused_norm_bf16_purego.go | 9 + .../kernels/fused_repeat_interleave_cgo.go | 6 + .../kernels/fused_repeat_interleave_purego.go | 21 ++ .../cuda/kernels/gemv_q4k_sm121_purego.go | 3 + internal/cuda/kernels/gemv_q5_0_purego.go | 3 + internal/cuda/kernels/gemv_q5k_purego.go | 3 + internal/cuda/kernels/gemv_q6k_purego.go | 3 + internal/cuda/kernels/offset_memcpy_purego.go | 6 + .../kernels/optional_symbol_guard_test.go | 281 ++++++++++++++++++ internal/cuda/kernels/purego.go | 13 +- internal/cuda/kernels/rope_select_purego.go | 3 + internal/cuda/kernels/sgemv_m1_purego.go | 3 + 18 files changed, 612 insertions(+), 1 deletion(-) create mode 100644 compute/gpu_repeat_interleave_parity_test.go create mode 100644 internal/cuda/kernels/optional_symbol_guard_test.go diff --git a/compute/gpu_repeat_interleave_parity_test.go b/compute/gpu_repeat_interleave_parity_test.go new file mode 100644 index 0000000..a66be51 --- /dev/null +++ b/compute/gpu_repeat_interleave_parity_test.go @@ -0,0 +1,193 @@ +package compute + +import ( + "context" + "testing" + + "github.com/zerfoo/ztensor/internal/cuda/kernels" + "github.com/zerfoo/ztensor/tensor" +) + +// ADR-091 harness 2 (engine parity) for the fused GQA head-expansion kernel, +// the op behind ztensor#180. +// +// Two hazards this file is written against, both live in this repo's history: +// +// 1. GPUEngine.RepeatInterleave falls back to the generic Reshape -> Repeat -> +// Reshape chain on ANY failure, including "the kernel is not in the +// deployed libkernels.so". A parity test that only compares values would +// therefore go green while the fused kernel never ran -- the exact vacuous +// gate this repo keeps rediscovering. So the test asserts the fused kernel +// is present FIRST, and fails rather than skips if it is not. +// +// 2. A head-expansion bug is a MAPPING bug: output head q must read KV head +// q/rep. If every KV head held similar data, a wrong mapping would still +// compare equal (docs/lore.md L-0009). So each element carries its own +// source coordinates, and the test proves that encoding is discriminating +// before it trusts the comparison. + +// riEncode returns a value that uniquely identifies the source coordinate and +// is exactly representable in float32, so the comparison below can be exact. +func riEncode(b, kv, s, d int) float32 { + return float32(((b*64+kv)*64+s)*64 + d) +} + +// riReference computes the expected [B, numKV*rep, S, D] output independently +// of any engine: output[b][q][s][d] = input[b][q/rep][s][d]. This is a closed +// form, not a second call into ztensor, so a mapping bug shared by Repeat and +// RepeatInterleave cannot hide behind it. +func riReference(b, numKV, s, d, rep int) []float32 { + numQ := numKV * rep + out := make([]float32, b*numQ*s*d) + i := 0 + for bi := 0; bi < b; bi++ { + for q := 0; q < numQ; q++ { + for si := 0; si < s; si++ { + for di := 0; di < d; di++ { + out[i] = riEncode(bi, q/rep, si, di) + i++ + } + } + } + } + return out +} + +func TestGPUEngine_RepeatInterleave_CPUParity(t *testing.T) { + gpuEng := newTestGPUEngine(t) // skips when CUDA is unavailable + ctx := context.Background() + + // Hazard 1. Not a skip: on the GB10 this test exists to prove the FUSED + // path works, and a green that silently measured the fallback is worse + // than a red. If this fires, rebuild libkernels.so from this tree + // (internal/cuda/kernels: make CUDA_ARCH=sm_121 shared) and redeploy it to + // the directory on LD_LIBRARY_PATH. + if !kernels.IsRepeatInterleaveF32Supported() { + t.Fatal("launch_repeat_interleave_f32 is not in the loaded libkernels.so; " + + "GPUEngine.RepeatInterleave would silently fall back and this test " + + "would prove nothing about the fused kernel (ztensor#180)") + } + + cases := []struct { + name string + B, numKV, S, D, rep int + }{ + // The head counts from the ztensor#180 repro (zerfoo TestGPUParity_GQA). + {"issue180_repro_1x2x2x8_rep2", 1, 2, 2, 8, 2}, + // rep=1 is the identity case: numQ == numKV. + {"identity_rep1", 2, 4, 3, 16, 1}, + // Llama-ish 32 query heads over 8 KV heads. + {"llama_8kv_rep4", 1, 8, 7, 64, 4}, + // Batch > 1 exercises the b term of the index decomposition, which a + // single-batch test cannot see at all. + {"batch3_rep3", 3, 2, 5, 32, 3}, + // Total elements not a multiple of the 256-thread block: exercises the + // kernel's `idx >= total` tail guard. + {"ragged_tail", 1, 3, 5, 7, 2}, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + in := make([]float32, tc.B*tc.numKV*tc.S*tc.D) + i := 0 + for b := 0; b < tc.B; b++ { + for kv := 0; kv < tc.numKV; kv++ { + for s := 0; s < tc.S; s++ { + for d := 0; d < tc.D; d++ { + in[i] = riEncode(b, kv, s, d) + i++ + } + } + } + } + + want := riReference(tc.B, tc.numKV, tc.S, tc.D, tc.rep) + + // Hazard 2, self-test: prove the reference actually distinguishes + // KV heads before trusting equality against it. If numKV > 1 the + // slice for output head 0 must differ from the slice for output + // head `rep`, which reads a DIFFERENT KV head. Without this, a + // degenerate encoding would make any mapping compare equal. + if tc.numKV > 1 { + stride := tc.S * tc.D + same := true + for j := 0; j < stride; j++ { + if want[j] != want[tc.rep*stride+j] { + same = false + break + } + } + if same { + t.Fatalf("reference is position-blind: output heads 0 and %d are "+ + "identical, so a wrong KV-head mapping could not be detected", tc.rep) + } + } + + a, err := tensor.New[float32]([]int{tc.B, tc.numKV, tc.S, tc.D}, in) + if err != nil { + t.Fatalf("tensor.New: %v", err) + } + + got, err := gpuEng.RepeatInterleave(ctx, a, 1, tc.rep) + if err != nil { + t.Fatalf("RepeatInterleave: %v", err) + } + + wantShape := []int{tc.B, tc.numKV * tc.rep, tc.S, tc.D} + gotShape := got.Shape() + if len(gotShape) != len(wantShape) { + t.Fatalf("shape rank: got %v want %v", gotShape, wantShape) + } + for j := range wantShape { + if gotShape[j] != wantShape[j] { + t.Fatalf("shape: got %v want %v", gotShape, wantShape) + } + } + + gd := got.Data() + if len(gd) != len(want) { + t.Fatalf("length: got %d want %d", len(gd), len(want)) + } + // Exact equality is the correct tolerance: repeat-interleave is a + // pure gather, no arithmetic is performed on the values, so any + // nonzero difference is a real defect. A float tolerance here + // would be the ztensor#182 mistake (a bound looser than the + // measured error, which can no longer fail). + mismatches := 0 + for j := range gd { + if gd[j] != want[j] { + if mismatches < 5 { + t.Errorf("element %d: got %v want %v", j, gd[j], want[j]) + } + mismatches++ + } + } + if mismatches > 0 { + t.Fatalf("%d/%d elements differ from the reference expansion", mismatches, len(gd)) + } + }) + } +} + +// TestGPUEngine_RepeatInterleave_UnsupportedFallsBack pins the contract the +// zerfoo caller relies on: for shapes/axes the fused kernel does not cover, +// RepeatInterleave must return a correct result via the generic path rather +// than error or crash. +func TestGPUEngine_RepeatInterleave_UnsupportedFallsBack(t *testing.T) { + gpuEng := newTestGPUEngine(t) + ctx := context.Background() + + // axis != 1 is outside the fused kernel's contract. + in := []float32{1, 2, 3, 4, 5, 6, 7, 8} + a, err := tensor.New[float32]([]int{1, 2, 2, 2}, in) + if err != nil { + t.Fatalf("tensor.New: %v", err) + } + got, err := gpuEng.RepeatInterleave(ctx, a, 2, 2) + if err != nil { + t.Fatalf("RepeatInterleave on the fallback path: %v", err) + } + if got.Size() != len(in)*2 { + t.Fatalf("fallback size: got %d want %d", got.Size(), len(in)*2) + } +} diff --git a/internal/cuda/kernels/counter_purego.go b/internal/cuda/kernels/counter_purego.go index ed78a8b..7ec683e 100644 --- a/internal/cuda/kernels/counter_purego.go +++ b/internal/cuda/kernels/counter_purego.go @@ -15,6 +15,9 @@ func IncrementCounter(counter unsafe.Pointer, delta int, s unsafe.Pointer) error if k == nil { return fmt.Errorf("increment_counter kernel: kernels not available") } + if k.launchIncrementCounter == 0 { + return fmt.Errorf("increment_counter kernel: launch_increment_counter not present in libkernels.so") + } ret := cuda.Ccall(k.launchIncrementCounter, uintptr(counter), uintptr(delta), uintptr(s)) return checkKernel(ret, "increment_counter") } @@ -25,6 +28,9 @@ func ResetCounter(counter unsafe.Pointer, value int, s unsafe.Pointer) error { if k == nil { return fmt.Errorf("reset_counter kernel: kernels not available") } + if k.launchResetCounter == 0 { + return fmt.Errorf("reset_counter kernel: launch_reset_counter not present in libkernels.so") + } ret := cuda.Ccall(k.launchResetCounter, uintptr(counter), uintptr(value), uintptr(s)) return checkKernel(ret, "reset_counter") } diff --git a/internal/cuda/kernels/elementwise_bf16_purego.go b/internal/cuda/kernels/elementwise_bf16_purego.go index 60b20d4..2003043 100644 --- a/internal/cuda/kernels/elementwise_bf16_purego.go +++ b/internal/cuda/kernels/elementwise_bf16_purego.go @@ -15,6 +15,9 @@ func AddBF16(a, b, c unsafe.Pointer, n int, s unsafe.Pointer) error { if k == nil { return fmt.Errorf("add_bf16 kernel: kernels not available") } + if k.launchAddBF16 == 0 { + return fmt.Errorf("add_bf16 kernel: launch_add_bf16 not present in libkernels.so") + } ret := cuda.Ccall(k.launchAddBF16, uintptr(a), uintptr(b), uintptr(c), uintptr(n), uintptr(s)) return checkKernel(ret, "add_bf16") } @@ -25,6 +28,9 @@ func SubBF16(a, b, c unsafe.Pointer, n int, s unsafe.Pointer) error { if k == nil { return fmt.Errorf("sub_bf16 kernel: kernels not available") } + if k.launchSubBF16 == 0 { + return fmt.Errorf("sub_bf16 kernel: launch_sub_bf16 not present in libkernels.so") + } ret := cuda.Ccall(k.launchSubBF16, uintptr(a), uintptr(b), uintptr(c), uintptr(n), uintptr(s)) return checkKernel(ret, "sub_bf16") } @@ -35,6 +41,9 @@ func MulBF16(a, b, c unsafe.Pointer, n int, s unsafe.Pointer) error { if k == nil { return fmt.Errorf("mul_bf16 kernel: kernels not available") } + if k.launchMulBF16 == 0 { + return fmt.Errorf("mul_bf16 kernel: launch_mul_bf16 not present in libkernels.so") + } ret := cuda.Ccall(k.launchMulBF16, uintptr(a), uintptr(b), uintptr(c), uintptr(n), uintptr(s)) return checkKernel(ret, "mul_bf16") } @@ -45,6 +54,9 @@ func DivBF16(a, b, c unsafe.Pointer, n int, s unsafe.Pointer) error { if k == nil { return fmt.Errorf("div_bf16 kernel: kernels not available") } + if k.launchDivBF16 == 0 { + return fmt.Errorf("div_bf16 kernel: launch_div_bf16 not present in libkernels.so") + } ret := cuda.Ccall(k.launchDivBF16, uintptr(a), uintptr(b), uintptr(c), uintptr(n), uintptr(s)) return checkKernel(ret, "div_bf16") } @@ -55,6 +67,9 @@ func TanhBF16(a, c unsafe.Pointer, n int, s unsafe.Pointer) error { if k == nil { return fmt.Errorf("tanh_bf16 kernel: kernels not available") } + if k.launchTanhBF16 == 0 { + return fmt.Errorf("tanh_bf16 kernel: launch_tanh_bf16 not present in libkernels.so") + } ret := cuda.Ccall(k.launchTanhBF16, uintptr(a), uintptr(c), uintptr(n), uintptr(s)) return checkKernel(ret, "tanh_bf16") } @@ -65,6 +80,9 @@ func SqrtBF16(a, c unsafe.Pointer, n int, s unsafe.Pointer) error { if k == nil { return fmt.Errorf("sqrt_bf16 kernel: kernels not available") } + if k.launchSqrtBF16 == 0 { + return fmt.Errorf("sqrt_bf16 kernel: launch_sqrt_bf16 not present in libkernels.so") + } ret := cuda.Ccall(k.launchSqrtBF16, uintptr(a), uintptr(c), uintptr(n), uintptr(s)) return checkKernel(ret, "sqrt_bf16") } @@ -75,6 +93,9 @@ func RsqrtBF16(a, c unsafe.Pointer, n int, s unsafe.Pointer) error { if k == nil { return fmt.Errorf("rsqrt_bf16 kernel: kernels not available") } + if k.launchRsqrtBF16 == 0 { + return fmt.Errorf("rsqrt_bf16 kernel: launch_rsqrt_bf16 not present in libkernels.so") + } ret := cuda.Ccall(k.launchRsqrtBF16, uintptr(a), uintptr(c), uintptr(n), uintptr(s)) return checkKernel(ret, "rsqrt_bf16") } @@ -85,6 +106,9 @@ func ExpBF16(a, c unsafe.Pointer, n int, s unsafe.Pointer) error { if k == nil { return fmt.Errorf("exp_bf16 kernel: kernels not available") } + if k.launchExpBF16 == 0 { + return fmt.Errorf("exp_bf16 kernel: launch_exp_bf16 not present in libkernels.so") + } ret := cuda.Ccall(k.launchExpBF16, uintptr(a), uintptr(c), uintptr(n), uintptr(s)) return checkKernel(ret, "exp_bf16") } @@ -95,6 +119,9 @@ func LogBF16(a, c unsafe.Pointer, n int, s unsafe.Pointer) error { if k == nil { return fmt.Errorf("log_bf16 kernel: kernels not available") } + if k.launchLogBF16 == 0 { + return fmt.Errorf("log_bf16 kernel: launch_log_bf16 not present in libkernels.so") + } ret := cuda.Ccall(k.launchLogBF16, uintptr(a), uintptr(c), uintptr(n), uintptr(s)) return checkKernel(ret, "log_bf16") } @@ -105,6 +132,9 @@ func F32ToBF16(src, dst unsafe.Pointer, n int, s unsafe.Pointer) error { if k == nil { return fmt.Errorf("f32_to_bf16 kernel: kernels not available") } + if k.launchF32ToBF16 == 0 { + return fmt.Errorf("f32_to_bf16 kernel: launch_f32_to_bf16 not present in libkernels.so") + } ret := cuda.Ccall(k.launchF32ToBF16, uintptr(src), uintptr(dst), uintptr(n), uintptr(s)) return checkKernel(ret, "f32_to_bf16") } @@ -115,6 +145,9 @@ func BF16ToF32(src, dst unsafe.Pointer, n int, s unsafe.Pointer) error { if k == nil { return fmt.Errorf("bf16_to_f32 kernel: kernels not available") } + if k.launchBF16ToF32 == 0 { + return fmt.Errorf("bf16_to_f32 kernel: launch_bf16_to_f32 not present in libkernels.so") + } ret := cuda.Ccall(k.launchBF16ToF32, uintptr(src), uintptr(dst), uintptr(n), uintptr(s)) return checkKernel(ret, "bf16_to_f32") } @@ -129,6 +162,9 @@ func SumAxisBF16(input, output unsafe.Pointer, outer, inner, axisSize int, invDi if k == nil { return fmt.Errorf("sum_axis_bf16 kernel: kernels not available") } + if k.launchSumAxisBF16 == 0 { + return fmt.Errorf("sum_axis_bf16 kernel: launch_sum_axis_bf16 not present in libkernels.so") + } ret := cuda.Ccall(k.launchSumAxisBF16, uintptr(input), uintptr(output), uintptr(outer), uintptr(inner), uintptr(axisSize), @@ -148,6 +184,9 @@ func ScaledSoftmaxBF16( if k == nil { return fmt.Errorf("scaled_softmax_bf16 kernel: kernels not available") } + if k.launchScaledSoftmaxBF16 == 0 { + return fmt.Errorf("scaled_softmax_bf16 kernel: launch_scaled_softmax_bf16 not present in libkernels.so") + } ret := cuda.Ccall(k.launchScaledSoftmaxBF16, uintptr(input), uintptr(output), uintptr(outer), uintptr(inner), uintptr(axisSize), diff --git a/internal/cuda/kernels/elementwise_fp16_purego.go b/internal/cuda/kernels/elementwise_fp16_purego.go index a0b2cee..08ae9ef 100644 --- a/internal/cuda/kernels/elementwise_fp16_purego.go +++ b/internal/cuda/kernels/elementwise_fp16_purego.go @@ -55,6 +55,9 @@ func F32ToFP16(src, dst unsafe.Pointer, n int, s unsafe.Pointer) error { if k == nil { return fmt.Errorf("f32_to_fp16 kernel: kernels not available") } + if k.launchF32ToFP16 == 0 { + return fmt.Errorf("f32_to_fp16 kernel: launch_f32_to_fp16 not present in libkernels.so") + } ret := cuda.Ccall(k.launchF32ToFP16, uintptr(src), uintptr(dst), uintptr(n), uintptr(s)) return checkKernel(ret, "f32_to_fp16") } @@ -65,6 +68,9 @@ func FP16ToF32(src, dst unsafe.Pointer, n int, s unsafe.Pointer) error { if k == nil { return fmt.Errorf("fp16_to_f32 kernel: kernels not available") } + if k.launchFP16ToF32 == 0 { + return fmt.Errorf("fp16_to_f32 kernel: launch_fp16_to_f32 not present in libkernels.so") + } ret := cuda.Ccall(k.launchFP16ToF32, uintptr(src), uintptr(dst), uintptr(n), uintptr(s)) return checkKernel(ret, "fp16_to_f32") } diff --git a/internal/cuda/kernels/fp8_ops_purego.go b/internal/cuda/kernels/fp8_ops_purego.go index 9a4f325..a524135 100644 --- a/internal/cuda/kernels/fp8_ops_purego.go +++ b/internal/cuda/kernels/fp8_ops_purego.go @@ -14,6 +14,9 @@ func DequantFP8E4M3ToFP16(input, output unsafe.Pointer, scale float32, n int, s if k == nil { return fmt.Errorf("dequant_fp8e4m3_to_fp16 kernel: kernels not available") } + if k.launchDequantFP8E4M3ToFP16 == 0 { + return fmt.Errorf("dequant_fp8e4m3_to_fp16 kernel: launch_dequant_fp8e4m3_to_fp16 not present in libkernels.so") + } ret := cuda.Ccall(k.launchDequantFP8E4M3ToFP16, uintptr(input), uintptr(output), floatBits(scale), uintptr(n), uintptr(s)) return checkKernel(ret, "dequant_fp8e4m3_to_fp16") @@ -26,6 +29,9 @@ func FP8Add(a, b, c unsafe.Pointer, scaleA, scaleB float32, n int, s unsafe.Poin if k == nil { return fmt.Errorf("fp8_add kernel: kernels not available") } + if k.launchFP8Add == 0 { + return fmt.Errorf("fp8_add kernel: launch_fp8_add not present in libkernels.so") + } ret := cuda.Ccall(k.launchFP8Add, uintptr(a), uintptr(b), uintptr(c), floatBits(scaleA), floatBits(scaleB), uintptr(n), uintptr(s)) @@ -39,6 +45,9 @@ func FP8Mul(a, b, c unsafe.Pointer, scaleA, scaleB float32, n int, s unsafe.Poin if k == nil { return fmt.Errorf("fp8_mul kernel: kernels not available") } + if k.launchFP8Mul == 0 { + return fmt.Errorf("fp8_mul kernel: launch_fp8_mul not present in libkernels.so") + } ret := cuda.Ccall(k.launchFP8Mul, uintptr(a), uintptr(b), uintptr(c), floatBits(scaleA), floatBits(scaleB), uintptr(n), uintptr(s)) @@ -53,6 +62,9 @@ func FP8RMSNorm(input, weight, output unsafe.Pointer, scale, eps float32, rows, if k == nil { return fmt.Errorf("fp8_rmsnorm kernel: kernels not available") } + if k.launchFP8RMSNorm == 0 { + return fmt.Errorf("fp8_rmsnorm kernel: launch_fp8_rmsnorm not present in libkernels.so") + } ret := cuda.Ccall(k.launchFP8RMSNorm, uintptr(input), uintptr(weight), uintptr(output), floatBits(scale), floatBits(eps), uintptr(rows), uintptr(D), uintptr(s)) diff --git a/internal/cuda/kernels/fused_adamw_bf16_purego.go b/internal/cuda/kernels/fused_adamw_bf16_purego.go index a68ff5f..686b760 100644 --- a/internal/cuda/kernels/fused_adamw_bf16_purego.go +++ b/internal/cuda/kernels/fused_adamw_bf16_purego.go @@ -23,6 +23,9 @@ func FusedAdamWBF16( if k == nil { return fmt.Errorf("fused_adamw_bf16 kernel: kernels not available") } + if k.launchFusedAdamWBF16 == 0 { + return fmt.Errorf("fused_adamw_bf16 kernel: fused_adamw_bf16 not present in libkernels.so") + } ret := cuda.Ccall(k.launchFusedAdamWBF16, uintptr(param), uintptr(m), uintptr(v), uintptr(grad), floatBitsF64(beta1), floatBitsF64(beta2), diff --git a/internal/cuda/kernels/fused_norm_bf16_purego.go b/internal/cuda/kernels/fused_norm_bf16_purego.go index 27fee0b..f099153 100644 --- a/internal/cuda/kernels/fused_norm_bf16_purego.go +++ b/internal/cuda/kernels/fused_norm_bf16_purego.go @@ -20,6 +20,9 @@ func FusedAddRMSNormBF16(input, residual, weight, normedOut, sumOut unsafe.Point if k == nil { return fmt.Errorf("fused_add_rmsnorm_bf16 kernel: kernels not available") } + if k.launchFusedAddRMSNormBF16 == 0 { + return fmt.Errorf("fused_add_rmsnorm_bf16 kernel: fused_add_rmsnorm_bf16 not present in libkernels.so") + } ret := cuda.Ccall(k.launchFusedAddRMSNormBF16, uintptr(input), uintptr(residual), uintptr(weight), uintptr(normedOut), uintptr(sumOut), floatBits(eps), uintptr(rows), uintptr(D), uintptr(s)) @@ -35,6 +38,9 @@ func FusedNormAddBF16(input, weight, residual, output unsafe.Pointer, eps float3 if k == nil { return fmt.Errorf("fused_norm_add_bf16 kernel: kernels not available") } + if k.launchFusedNormAddBF16 == 0 { + return fmt.Errorf("fused_norm_add_bf16 kernel: fused_norm_add_bf16 not present in libkernels.so") + } ret := cuda.Ccall(k.launchFusedNormAddBF16, uintptr(input), uintptr(weight), uintptr(residual), uintptr(output), floatBits(eps), uintptr(rows), uintptr(D), uintptr(s)) @@ -52,6 +58,9 @@ func FusedQKNormRoPEBF16(input, weightQ, weightK, cosAngles, sinAngles, output u if k == nil { return fmt.Errorf("fused_qk_norm_rope_bf16 kernel: kernels not available") } + if k.launchFusedQKNormRoPEBF16 == 0 { + return fmt.Errorf("fused_qk_norm_rope_bf16 kernel: fused_qk_norm_rope_bf16 not present in libkernels.so") + } ret := cuda.Ccall(k.launchFusedQKNormRoPEBF16, uintptr(input), uintptr(weightQ), uintptr(weightK), uintptr(cosAngles), uintptr(sinAngles), uintptr(output), floatBits(eps), diff --git a/internal/cuda/kernels/fused_repeat_interleave_cgo.go b/internal/cuda/kernels/fused_repeat_interleave_cgo.go index 4a37fca..76c6997 100644 --- a/internal/cuda/kernels/fused_repeat_interleave_cgo.go +++ b/internal/cuda/kernels/fused_repeat_interleave_cgo.go @@ -13,6 +13,12 @@ import ( "unsafe" ) +// IsRepeatInterleaveF32Supported reports whether the fused GQA head-expansion +// kernel is available. Under the cuda build tag the launcher is resolved by the +// linker at build time, so it is always present. The purego build resolves it +// with dlsym and can legitimately answer false (ztensor#180). +func IsRepeatInterleaveF32Supported() bool { return true } + // RepeatInterleaveF32 expands [B, numKV, S, D] to [B, numQ, S, D] for GQA head expansion. func RepeatInterleaveF32( input, output unsafe.Pointer, diff --git a/internal/cuda/kernels/fused_repeat_interleave_purego.go b/internal/cuda/kernels/fused_repeat_interleave_purego.go index e335690..737086c 100644 --- a/internal/cuda/kernels/fused_repeat_interleave_purego.go +++ b/internal/cuda/kernels/fused_repeat_interleave_purego.go @@ -9,6 +9,20 @@ import ( "github.com/zerfoo/ztensor/internal/cuda" ) +// IsRepeatInterleaveF32Supported reports whether the LOADED libkernels.so +// actually exports launch_repeat_interleave_f32. +// +// The symbol is optional (see purego.go optionalSyms), so a deployed library +// built before the kernel existed resolves it to 0. Callers that want the +// fused GQA head expansion -- and tests that mean to PROVE the fused path ran +// rather than a fallback -- must consult this instead of inferring support +// from a nil error, because compute.GPUEngine.RepeatInterleave silently +// degrades to the generic Reshape -> Repeat -> Reshape chain on any failure. +func IsRepeatInterleaveF32Supported() bool { + k := klib() + return k != nil && k.launchRepeatInterleaveF32 != 0 +} + // RepeatInterleaveF32 expands [B, numKV, S, D] to [B, numQ, S, D] for GQA head expansion. // Each KV head is repeated `rep` times along the head dimension (numQ = numKV * rep). func RepeatInterleaveF32( @@ -20,6 +34,13 @@ func RepeatInterleaveF32( if k == nil { return fmt.Errorf("repeat_interleave_f32 kernel: kernels not available") } + // ztensor#180: launch_repeat_interleave_f32 is an OPTIONAL symbol. Without + // this check a libkernels.so that predates the kernel leaves the pointer at + // 0 and cuda.Ccall jumps to address 0 -- SIGSEGV PC=0x0, which kills the + // process before any caller fallback can run. + if k.launchRepeatInterleaveF32 == 0 { + return fmt.Errorf("repeat_interleave_f32 kernel: launch_repeat_interleave_f32 not present in libkernels.so") + } ret := cuda.Ccall(k.launchRepeatInterleaveF32, uintptr(input), uintptr(output), uintptr(B), uintptr(numKV), uintptr(S), uintptr(D), uintptr(rep), diff --git a/internal/cuda/kernels/gemv_q4k_sm121_purego.go b/internal/cuda/kernels/gemv_q4k_sm121_purego.go index 048f6cb..ec7008f 100644 --- a/internal/cuda/kernels/gemv_q4k_sm121_purego.go +++ b/internal/cuda/kernels/gemv_q4k_sm121_purego.go @@ -47,6 +47,9 @@ func GemvQ4KSm121F32( if k == nil { return fmt.Errorf("gemv_q4k_sm121_f32 kernel: kernels not available") } + if k.launchGemvQ4KSm121F32 == 0 { + return fmt.Errorf("gemv_q4k_sm121_f32 kernel: gemv_q4k_sm121_f32 not present in libkernels.so") + } ret := cuda.Ccall(k.launchGemvQ4KSm121F32, uintptr(W_q4k), uintptr(x), uintptr(y), uintptr(M), uintptr(K), uintptr(stream)) diff --git a/internal/cuda/kernels/gemv_q5_0_purego.go b/internal/cuda/kernels/gemv_q5_0_purego.go index b6d1469..bc5812b 100644 --- a/internal/cuda/kernels/gemv_q5_0_purego.go +++ b/internal/cuda/kernels/gemv_q5_0_purego.go @@ -21,6 +21,9 @@ func GemvQ5_0F32( if k == nil { return fmt.Errorf("gemv_q5_0_f32 kernel: kernels not available") } + if k.launchGemvQ5_0F32 == 0 { + return fmt.Errorf("gemv_q5_0_f32 kernel: gemv_q5_0_f32 not present in libkernels.so") + } ret := cuda.Ccall(k.launchGemvQ5_0F32, uintptr(W_q5_0), uintptr(x), uintptr(y), uintptr(M), uintptr(K), diff --git a/internal/cuda/kernels/gemv_q5k_purego.go b/internal/cuda/kernels/gemv_q5k_purego.go index 60f4ba0..cdb809d 100644 --- a/internal/cuda/kernels/gemv_q5k_purego.go +++ b/internal/cuda/kernels/gemv_q5k_purego.go @@ -20,6 +20,9 @@ func GemvQ5KF32( if k == nil { return fmt.Errorf("gemv_q5k_f32 kernel: kernels not available") } + if k.launchGemvQ5KF32 == 0 { + return fmt.Errorf("gemv_q5k_f32 kernel: gemv_q5k_f32 not present in libkernels.so") + } ret := cuda.Ccall(k.launchGemvQ5KF32, uintptr(W_q5k), uintptr(x), uintptr(y), uintptr(M), uintptr(K), uintptr(stream)) diff --git a/internal/cuda/kernels/gemv_q6k_purego.go b/internal/cuda/kernels/gemv_q6k_purego.go index f43e987..ea15c48 100644 --- a/internal/cuda/kernels/gemv_q6k_purego.go +++ b/internal/cuda/kernels/gemv_q6k_purego.go @@ -20,6 +20,9 @@ func GemvQ6KF32( if k == nil { return fmt.Errorf("gemv_q6k_f32 kernel: kernels not available") } + if k.launchGemvQ6KF32 == 0 { + return fmt.Errorf("gemv_q6k_f32 kernel: gemv_q6k_f32 not present in libkernels.so") + } ret := cuda.Ccall(k.launchGemvQ6KF32, uintptr(W_q6k), uintptr(x), uintptr(y), uintptr(M), uintptr(K), uintptr(stream)) diff --git a/internal/cuda/kernels/offset_memcpy_purego.go b/internal/cuda/kernels/offset_memcpy_purego.go index 20751b8..4f322e8 100644 --- a/internal/cuda/kernels/offset_memcpy_purego.go +++ b/internal/cuda/kernels/offset_memcpy_purego.go @@ -16,6 +16,9 @@ func OffsetMemcpy(dst, src, counter unsafe.Pointer, dim, maxSeqLen int, s unsafe if k == nil { return fmt.Errorf("offset_memcpy kernel: kernels not available") } + if k.launchOffsetMemcpy == 0 { + return fmt.Errorf("offset_memcpy kernel: launch_offset_memcpy not present in libkernels.so") + } ret := cuda.Ccall(k.launchOffsetMemcpy, uintptr(dst), uintptr(src), uintptr(counter), uintptr(dim), uintptr(maxSeqLen), uintptr(s)) @@ -29,6 +32,9 @@ func OffsetMemcpyFP16(dst, src, counter unsafe.Pointer, dim, maxSeqLen int, s un if k == nil { return fmt.Errorf("offset_memcpy_fp16 kernel: kernels not available") } + if k.launchOffsetMemcpyFP16 == 0 { + return fmt.Errorf("offset_memcpy_fp16 kernel: launch_offset_memcpy_fp16 not present in libkernels.so") + } ret := cuda.Ccall(k.launchOffsetMemcpyFP16, uintptr(dst), uintptr(src), uintptr(counter), uintptr(dim), uintptr(maxSeqLen), uintptr(s)) diff --git a/internal/cuda/kernels/optional_symbol_guard_test.go b/internal/cuda/kernels/optional_symbol_guard_test.go new file mode 100644 index 0000000..78d1c94 --- /dev/null +++ b/internal/cuda/kernels/optional_symbol_guard_test.go @@ -0,0 +1,281 @@ +//go:build !cuda + +package kernels + +import ( + "go/ast" + "go/parser" + "go/token" + "os" + "path/filepath" + "regexp" + "sort" + "strings" + "testing" + "unsafe" +) + +// --- ztensor#180 regression ------------------------------------------------- +// +// openKernelLib treats a set of kernel symbols as OPTIONAL: when dlsym cannot +// find one in the deployed libkernels.so it leaves the function pointer at 0 +// and continues, on the contract (purego.go) that "callers check before use". +// +// RepeatInterleaveF32 did not check. On the GB10 the deployed +// /opt/zerfoo/lib/libkernels.so does not export launch_repeat_interleave_f32, +// so the wrapper called cuda.Ccall(0, ...) -- a jump to address 0. That is a +// SIGSEGV with PC=0x0 inside cgo, which kills the process outright: it is not +// an error return, so no caller fallback can run. Every GQA model on GPU +// (Llama, Mistral, Qwen, Gemma) died there. See ztensor#180. + +// stubKernelLib makes klib() return lib for the duration of the test. +// +// It deliberately forces the real load FIRST so that kernelLibOnce is consumed +// by the genuine dlopen path, never by this stub -- otherwise a test running +// after this one would silently see "no kernel lib" on a machine that has one. +func stubKernelLib(t *testing.T, lib *KernelLib) { + t.Helper() + _, _ = openKernelLib() + prevLib, prevErr := kernelLib, errKernelLib + kernelLib, errKernelLib = lib, nil + t.Cleanup(func() { kernelLib, errKernelLib = prevLib, prevErr }) +} + +// TestRepeatInterleaveF32_UnresolvedSymbol_ReturnsError is the ztensor#180 +// regression. With a KernelLib whose launchRepeatInterleaveF32 is 0 -- exactly +// the state produced by a libkernels.so that predates the kernel -- the +// wrapper must return an error. Before the fix this test does not fail, it +// CRASHES the test binary with SIGSEGV PC=0x0. +func TestRepeatInterleaveF32_UnresolvedSymbol_ReturnsError(t *testing.T) { + // handle is non-zero: the library loaded fine, only the optional symbol + // is missing. This is the real-world state, and it is what makes the + // k == nil check insufficient. + stubKernelLib(t, &KernelLib{handle: 1}) + + // Non-nil, non-null buffers: the issue's original hypothesis was an + // unallocated dst, so pass plainly valid pointers to rule that out. The + // only null in play is the function pointer. + in := make([]float32, 1*2*2*8) + out := make([]float32, 1*4*2*8) + + err := RepeatInterleaveF32( + unsafe.Pointer(&in[0]), unsafe.Pointer(&out[0]), + 1, 2, 2, 8, 2, + nil, + ) + if err == nil { + t.Fatal("RepeatInterleaveF32 returned nil error with an unresolved kernel symbol; " + + "the caller fallback can never engage") + } + // Sensitivity: the error must name the missing SYMBOL, not merely report + // that kernels are unavailable. A blanket "kernels not available" would + // satisfy err != nil while hiding which of the two distinct failures + // occurred, and would pass even if the guard were removed again and + // replaced with an unconditional error. + if !strings.Contains(err.Error(), "launch_repeat_interleave_f32") { + t.Fatalf("error does not identify the unresolved symbol: %v", err) + } +} + +// TestRepeatInterleaveF32_NoKernelLib_ReturnsDistinctError is the other half of +// the sensitivity pair: when the library itself is absent the wrapper must +// report THAT, not the missing-symbol condition. Two distinguishable errors +// prove the guard discriminates rather than failing blanket. +func TestRepeatInterleaveF32_NoKernelLib_ReturnsDistinctError(t *testing.T) { + stubKernelLib(t, nil) + + in := make([]float32, 8) + out := make([]float32, 16) + err := RepeatInterleaveF32( + unsafe.Pointer(&in[0]), unsafe.Pointer(&out[0]), + 1, 1, 1, 8, 2, + nil, + ) + if err == nil { + t.Fatal("RepeatInterleaveF32 returned nil error with no kernel library") + } + if strings.Contains(err.Error(), "launch_repeat_interleave_f32") { + t.Fatalf("no-library case reported as a missing-symbol case: %v", err) + } +} + +// --- class gate ------------------------------------------------------------- + +var ( + symBindingRe = regexp.MustCompile(`\{"([A-Za-z0-9_]+)",\s*&k\.([A-Za-z0-9_]+)\}`) + optionalKeyRe = regexp.MustCompile(`^\s*"([A-Za-z0-9_]+)":\s*true,`) +) + +// loadSymbolTables parses purego.go for the symbol -> struct-field bindings and +// for the optionalSyms key set. +func loadSymbolTables(t *testing.T) (fieldForSym map[string]string, optional map[string]bool) { + t.Helper() + src, err := os.ReadFile("purego.go") + if err != nil { + t.Fatalf("read purego.go: %v", err) + } + text := string(src) + + fieldForSym = map[string]string{} + for _, m := range symBindingRe.FindAllStringSubmatch(text, -1) { + fieldForSym[m[1]] = m[2] + } + + optional = map[string]bool{} + inBlock := false + for _, line := range strings.Split(text, "\n") { + switch { + case strings.Contains(line, "optionalSyms := map"): + inBlock = true + case inBlock && strings.TrimSpace(line) == "}": + inBlock = false + case inBlock: + if m := optionalKeyRe.FindStringSubmatch(line); m != nil { + optional[m[1]] = true + } + } + } + + // Anti-vacuous: if either extraction silently returned nothing (a format + // change in purego.go), this gate would pass while checking nothing. + if len(fieldForSym) < 100 { + t.Fatalf("symbol/field extraction found only %d bindings; parser is out of "+ + "sync with purego.go and this gate is not checking anything", len(fieldForSym)) + } + if len(optional) < 20 { + t.Fatalf("optionalSyms extraction found only %d entries; parser is out of "+ + "sync with purego.go", len(optional)) + } + return fieldForSym, optional +} + +// TestOptionalKernelSymbolsAreGuarded enforces the contract stated at +// purego.go's optionalSyms ("callers check before use") across the whole +// package: every cuda.Ccall whose function pointer is an OPTIONAL kernel +// symbol must sit in a function that first compares that pointer to 0. +// +// This is the class gate for ztensor#180. RepeatInterleaveF32 was not a +// one-off: an optional symbol is exactly the symbol most likely to be absent +// from a deployed libkernels.so, and an unguarded launch of it is an +// unrecoverable process kill rather than a fallback-able error. +func TestOptionalKernelSymbolsAreGuarded(t *testing.T) { + fieldForSym, optional := loadSymbolTables(t) + + optionalField := map[string]string{} // field -> symbol name + for sym := range optional { + f, ok := fieldForSym[sym] + if !ok { + t.Errorf("optionalSyms lists %q but no {%q, &k.Field} binding exists", sym, sym) + continue + } + optionalField[f] = sym + } + + files, err := filepath.Glob("*.go") + if err != nil { + t.Fatalf("glob: %v", err) + } + fset := token.NewFileSet() + + type violation struct{ file, fn, field, sym string } + var violations []violation + callSites := 0 + + for _, file := range files { + if strings.HasSuffix(file, "_test.go") || file == "purego.go" { + continue + } + f, perr := parser.ParseFile(fset, file, nil, 0) + if perr != nil { + t.Fatalf("parse %s: %v", file, perr) + } + for _, decl := range f.Decls { + fn, ok := decl.(*ast.FuncDecl) + if !ok || fn.Body == nil { + continue + } + guarded := guardedFields(fn.Body) + for _, field := range ccallFields(fn.Body) { + callSites++ + sym, isOptional := optionalField[field] + if !isOptional || guarded[field] { + continue + } + violations = append(violations, violation{file, fn.Name.Name, field, sym}) + } + } + } + + // Anti-vacuous: a walker that matches no call sites proves nothing. + if callSites == 0 { + t.Fatal("found no cuda.Ccall(k., ...) call sites; the AST walker is " + + "not matching and this gate is not checking anything") + } + t.Logf("checked %d cuda.Ccall sites against %d optional symbols", callSites, len(optionalField)) + + sort.Slice(violations, func(i, j int) bool { return violations[i].sym < violations[j].sym }) + for _, v := range violations { + t.Errorf("%s: %s launches optional symbol %s (k.%s) without checking it is non-zero; "+ + "a missing symbol makes this a SIGSEGV, not an error (ztensor#180)", + v.file, v.fn, v.sym, v.field) + } +} + +// ccallFields returns the KernelLib field names used as the function-pointer +// argument of cuda.Ccall within body. +func ccallFields(body *ast.BlockStmt) []string { + var out []string + ast.Inspect(body, func(n ast.Node) bool { + call, ok := n.(*ast.CallExpr) + if !ok || len(call.Args) == 0 { + return true + } + sel, ok := call.Fun.(*ast.SelectorExpr) + if !ok || sel.Sel.Name != "Ccall" { + return true + } + if pkg, ok := sel.X.(*ast.Ident); !ok || pkg.Name != "cuda" { + return true + } + if field, ok := kernelLibField(call.Args[0]); ok { + out = append(out, field) + } + return true + }) + return out +} + +// guardedFields returns the KernelLib field names compared against 0 anywhere +// in body. +func guardedFields(body *ast.BlockStmt) map[string]bool { + out := map[string]bool{} + ast.Inspect(body, func(n ast.Node) bool { + bin, ok := n.(*ast.BinaryExpr) + if !ok || (bin.Op != token.EQL && bin.Op != token.NEQ) { + return true + } + for _, pair := range [2][2]ast.Expr{{bin.X, bin.Y}, {bin.Y, bin.X}} { + field, isField := kernelLibField(pair[0]) + lit, isLit := pair[1].(*ast.BasicLit) + if isField && isLit && lit.Kind == token.INT && lit.Value == "0" { + out[field] = true + } + } + return true + }) + return out +} + +// kernelLibField reports whether e is a selector on the local KernelLib +// variable (conventionally `k`), returning the field name. +func kernelLibField(e ast.Expr) (string, bool) { + sel, ok := e.(*ast.SelectorExpr) + if !ok { + return "", false + } + recv, ok := sel.X.(*ast.Ident) + if !ok || recv.Name != "k" { + return "", false + } + return sel.Sel.Name, true +} diff --git a/internal/cuda/kernels/purego.go b/internal/cuda/kernels/purego.go index 928adee..82ecdfb 100644 --- a/internal/cuda/kernels/purego.go +++ b/internal/cuda/kernels/purego.go @@ -425,7 +425,18 @@ func openKernelLib() (*KernelLib, error) { {"fused_encoder_fwd_scratch_bytes", &k.launchFusedEncoderFwdScratch}, {"fused_encoder_bwd_f32", &k.launchFusedEncoderBwdF32}, } - // Optional symbols: missing is non-fatal (kernel not compiled yet). + // Optional symbols: missing is non-fatal (kernel not compiled yet, or -- + // the common case -- the DEPLOYED libkernels.so predates it; the GB10 + // loads a prebuilt /opt/zerfoo/lib/libkernels.so, not a build of this + // tree). + // + // A missing optional symbol leaves its function pointer at 0. Every + // wrapper MUST compare its pointer to 0 before cuda.Ccall: calling + // through 0 is a jump to address 0, i.e. SIGSEGV with PC=0x0 inside + // cgo, which kills the process instead of returning an error a caller + // could fall back from. That is exactly how ztensor#180 took down every + // GQA model on GPU. TestOptionalKernelSymbolsAreGuarded enforces this + // across the package. optionalSyms := map[string]bool{ "gemv_q4k_dp4a_f32": true, "gemv_q4k_sm121_f32": true, // sm_121 optimized; requires Blackwell diff --git a/internal/cuda/kernels/rope_select_purego.go b/internal/cuda/kernels/rope_select_purego.go index 87ae86a..3378817 100644 --- a/internal/cuda/kernels/rope_select_purego.go +++ b/internal/cuda/kernels/rope_select_purego.go @@ -17,6 +17,9 @@ func RoPESelect(cosTable, sinTable, cosOut, sinOut, counter unsafe.Pointer, if k == nil { return fmt.Errorf("rope_select kernel: kernels not available") } + if k.launchRoPESelect == 0 { + return fmt.Errorf("rope_select kernel: launch_rope_select not present in libkernels.so") + } ret := cuda.Ccall(k.launchRoPESelect, uintptr(cosTable), uintptr(sinTable), uintptr(cosOut), uintptr(sinOut), diff --git a/internal/cuda/kernels/sgemv_m1_purego.go b/internal/cuda/kernels/sgemv_m1_purego.go index 9d22407..ce02769 100644 --- a/internal/cuda/kernels/sgemv_m1_purego.go +++ b/internal/cuda/kernels/sgemv_m1_purego.go @@ -16,6 +16,9 @@ func SgemvM1(y, A, x unsafe.Pointer, M, N int, s unsafe.Pointer) error { if k == nil { return fmt.Errorf("sgemv_m1 kernel: kernels not available") } + if k.launchSgemvM1 == 0 { + return fmt.Errorf("sgemv_m1 kernel: launch_sgemv_m1 not present in libkernels.so") + } ret := cuda.Ccall(k.launchSgemvM1, uintptr(y), uintptr(A), uintptr(x), uintptr(M), uintptr(N), uintptr(s)) From c29246e8ac17d3dda4b0266bbed5fd219f300e0b Mon Sep 17 00:00:00 2001 From: David Ndungu Date: Sun, 23 Aug 2026 16:50:01 -0700 Subject: [PATCH 2/2] feat(compute): expose FusedRepeatInterleaveAvailable RepeatInterleave falls back to the generic Reshape -> Repeat -> Reshape chain on any failure and returns a correct result either way, so callers cannot tell from its output whether the fused kernel ran. zerfoo needs that distinction to assert its restored fused GQA path is genuinely fused rather than quietly measuring the fallback, and kernels.IsRepeatInterleaveF32Supported lives under internal/ where zerfoo cannot reach it. Mirrors the existing kernels.IsPagedAttentionSupported usage in compute/gpu_paged_gqa.go. --- compute/gpu_engine_memory.go | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/compute/gpu_engine_memory.go b/compute/gpu_engine_memory.go index 14e5aee..e9db696 100644 --- a/compute/gpu_engine_memory.go +++ b/compute/gpu_engine_memory.go @@ -10,6 +10,7 @@ import ( "os" "unsafe" + "github.com/zerfoo/ztensor/internal/cuda/kernels" "github.com/zerfoo/ztensor/internal/gpuapi" "github.com/zerfoo/ztensor/tensor" ) @@ -633,6 +634,19 @@ func (e *GPUEngine[T]) Repeat(ctx context.Context, a *tensor.TensorNumeric[T], a return makeGPUResult[T](e, newShape, devOut, outElems, dst...) } +// FusedRepeatInterleaveAvailable reports whether RepeatInterleave can actually +// use the fused kernel, as opposed to silently degrading to the generic +// Reshape -> Repeat -> Reshape chain. +// +// RepeatInterleave falls back on ANY failure and returns a correct result +// either way, so a caller (or a test) that only inspects its output cannot +// tell the two apart. Anything that claims the fused path is in use -- a +// benchmark, a parity gate -- must consult this, or it is measuring something +// it cannot name. See ztensor#180. +func (e *GPUEngine[T]) FusedRepeatInterleaveAvailable() bool { + return isFloat32[T]() && kernels.IsRepeatInterleaveF32Supported() +} + // RepeatInterleave expands a 4D tensor from [B, numKV, S, D] to [B, numQ, S, D] // by repeating each head along axis 1 (the head dimension) `reps` times. // This is a fused kernel for GQA key/value head expansion, replacing the