feat: TurboQuant KV-cache quantization — all backends - #1
Merged
Merged
Conversation
Port of the TurboQuant codec (arXiv 2504.19874, reference: TheTom/llama-cpp-turboquant) onto this fork's base: WHT-rotated polar quantization for the KV cache. - new types TURBO2_0/TURBO3_0/TURBO4_0 (KV, 2/3/4-bit blocks of 128) and TQ3_1S/TQ4_1S (WHT-rotated Lloyd-Max weight formats), renumbered for this tree's type table - GGML_OP_TURBO_WHT + ggml_turbo_wht() builder (forward/inverse Walsh-Hadamard rotation with sign arrays, InnerQ scale hook) - ggml-turbo-quant.c codec core, rotation data tables, CPU op + vec_dot paths, type traits, quantize-chunk wiring - CUDA/Metal/Vulkan kernel sources imported (wired in follow-ups) - verified: test-turbo-quant round-trip on CPU (cosine 0.986-0.996)
- pre-rotate Q (forward WHT) at every cache-backed attention site when K is a turbo type; no-op otherwise - inverse WHT on attention output in both FA and non-FA paths of build_attn_mha (group size derived from K, contiguity enforced) - register turbo2/3/4 as --cache-type-k/--cache-type-v values Core scope: no InnerQ scaling, no head-dim padding — turbo KV requires 128-aligned (or 64-group) head dims
Wire the TurboQuant Metal backend host side and kernels on top of the CPU core already on this branch. Metal kernels (ggml-metal.metal): - Inlined turbo-wht.h (WHT sign arrays incl. half4 variants + FWHT butterfly + rotate fwd/inverse) and the turbo2/3/4 centroids. - quantize/dequantize_turbo2_0/3_0/4_0 (+_t4) and full-block cache. - kernel_turbo_wht (O(d log d) Walsh-Hadamard rotation op) and kernel_turbo4_dequant_f16 bulk prefill dequant. - kernel_set_rows_turbo/2/4 cache-quantize kernels + instantiations. - flash_attn_ext / flash_attn_ext_vec instantiations for symmetric and asymmetric turbo2/3/4 and q8_0<->turbo pairs. - TurboFlash two-pass kernels (p1/p2), default-disabled via env. Host side: - ggml-metal-device.cpp: get_pipeline_turbo_wht; flash-attn pipeline base names now encode both K and V types (k%s_v%s) so asymmetric K/V pairs resolve unambiguously. All existing FA kernel host_names in the shader renamed to match. - ggml-metal-device.m: TurboQuant compile-time env prep (4-mag LUT auto select, sparse-V, profile mode); supports_op accepts turbo K types and asymmetric turbo/q8_0 FA pairs; GGML_OP_TURBO_WHT. - ggml-metal-ops.cpp: GGML_OP_TURBO_WHT encode case + impl; asymmetric K/V assert relaxation; TurboFlash gating (use_turbo_flash, extra_tmp sizing, force-nonvec) + two-pass dispatch; +128 float FA smem for the turbo block cache. - ggml-metal-impl.h / -device.h / -ops.h: kargs (turbo_wht, turbo_flash_p1/p2), FC offsets, pipeline/op declarations. Adaptations vs the reference tree (older upstream vintage): - Reference intertwines separate TQ3_1S/TQ4_1S weight-quant work; only the turbo KV-cache lines are ported here. - set_rows_turbo dispatch selection is not wired: our upstream refactored get_pipeline_set_rows/kernel_set_rows_q32 to a different signature, and it is out of scope for this KV port. supports_op therefore does not claim CPY/SET_ROWS turbo support, so cache writes fall back to CPU; the set_rows_turbo shader kernels are present for when dispatch lands. - No InnerQ scale tensors touched (graph passes nullptr scale).
…patch, dequant) Port the TurboQuant CUDA backend from the reference implementation, adapted to this newer upstream vintage: - set-rows.cu: k_set_rows_turbo2/3/4 kernels (WHT rotation + norm-corrected PolarQuant KV write, InnerQ calibration hooks) + dispatch. turbo4 rnorm write dropped (field removed from block_turbo4_0 on this branch). - dequantize.cuh: dequantize_turbo2/3/4_0 and dequantize_tq3_1s/tq4_1s device functions. - convert.cu: turbo + TQ dequant cases in the fp16/fp32 contiguous and non-contiguous dispatch tables; warp-cooperative TQ4_1S dequant kernel. - fattn-common.cuh: vec_dot_fattn_vec_KQ_turbo2/3/4_0, dequantize_V_turbo* and get_vec_dot_KQ / get_dequantize_V dispatch entries. - fattn-vec.cuh: turbo K types take the float-Q path (Q_q8_1 off), turbo V uses the ne=4 quantized dequant path; extern decls for the 21 compiled K/V type combinations. Reference-side perf extras (shared-mem centroid LUT, sparse-V skip, nthreads tuning) intentionally not ported. - fattn.cu: turbo types accepted in kv_type_supported, vec kernel cases registered. Decode goes through the VEC kernel; prefill converts KV to f16 via the ported nc dequant path. - fattn-mma-turbo.cuh: reduced to a stub launcher. No typed MMA kernel exists in the reference either; the pre-existing template instances now compile against the stub and the dispatcher never selects this path. - ggml-cuda.cu: GGML_OP_TURBO_WHT dispatch + supports_op, SET_ROWS turbo gates (head_dim % 64, turbo4 % 128). - CMakeLists.txt: turbo fattn-vec instances added to the non-FA_ALL_QUANTS build. Verified on RTX 3060 (sm_86) with Bonsai-27B-Q1_0, partial offload, fa on: coherent generation for turbo2/3/4; perplexity over 4 chunks: f16 KV 7.9486 +/- 0.73, turbo3 KV 8.0261 +/- 0.74 (CUDA) vs 8.0279 (CPU-only).
…ispatch, WHT op) - register turbo_wht.comp and set_rows turbo2/3/4 quantize variants in vulkan-shaders-gen; add dedicated DATA_A_TURBO3_0 flash-attn SPIR-V (scalar + coopmat1) with turbo3-only K/V bindings - copy_to_quant: 128-thread per-block set_rows path (normalize, randomized WHT, Lloyd-Max centroid quantize, subgroup pack), shared WHT tables across the three turbo widths - flash_attn_dequant: turbo2/3/4 centroid dequant macros wired into the FaTypeK/FaTypeV runtime switch (ids 43/44/45); f16 K/V aliases guarded out under DATA_A_TURBO3_0 - ggml-vulkan: GGML_OP_TURBO_WHT pipeline + dispatch, turbo set_rows pipelines (required subgroup size 32), FA pipeline selection for turbo3, supports_op gates (128-elem blocks, turbo3 K/V symmetry, scalar/coopmat1 only — cm2 falls back to scalar)
CI builds with -Werror: defining the GGML_API-decorated global with an initializer warns on every platform. Plain definition in the C TU; the C++ consumer now declares it extern "C" (it previously referenced a mangled symbol).
- generalize the dedicated turbo flash-attn SPIR-V path from turbo3-only
to all three tiers: DATA_A_TURBO_KV umbrella define in types.glsl gates
out the f16/q4/q5/q8 K/V binding aliases in flash_attn.comp,
flash_attn_cm1.comp and flash_attn_dequant.glsl for any turbo tier
- flash_attn_dequant: per-compilation dequantize4 for DATA_A_TURBO2_0
(2-bit centroids, fp16 norm + 32B idx) and DATA_A_TURBO4_0 (4-bit
nibble idx, fp16 norm + fp16 aux + 64B) alongside the turbo3 one
- vulkan-shaders-gen: register flash_attn_f32_f16_turbo{2,4}_0 scalar +
coopmat1 variants via a per-tier loop
- ggml-vulkan: FA pipeline selection picks the matching turbo SPIR-V for
turbo2/turbo4 k_type (scalar + cm1), supports_op requires symmetric
K/V per tier (turbo2/turbo2, turbo3/turbo3, turbo4/turbo4)
- map cudaMemcpyToSymbol/FromSymbol and __ballot_sync in vendor headers - pass explicit width to __shfl*_sync (HIP macros take 4 args) - drop unused warp_id locals flagged by nvcc -Werror
… quantize test - tmp_q2 overflowed for types with vec_dot_type F32 (4 bytes per element) - skip turbo KV types: they quantize in the rotated domain, round trip does not reproduce the input by design - tq3_1s uses the 3-bit and lowbit thresholds
The gfx908 build of the ncols=2 turbo3 vec kernels lands at 259 total VGPRs, slightly over the 256 gate. Register them like the other known high-pressure fattn kernels.
BAIS1C
pushed a commit
to BAIS1C/llama.cpp
that referenced
this pull request
Aug 3, 2026
* Sycl tp stage1 (thecodacus#1) * SYCL: tensor parallelism (--split-mode tensor) for dual-GPU Adds the comm_init/comm_free/comm_allreduce_tensor trio that the meta-backend queries via get_proc_address to enable backend-specific all-reduce, mirroring the pattern used by ggml-cuda.cu. For N=2 (the common dual-GPU case) implements a degenerate ring all-reduce with two size-branched paths: * Small (nelem < 32768): FP32 direct memcpy + per-device ADD kernel chained via depends_on(memcpy_event). 4 SYCL submissions/call. * Large (nelem >= 32768): BF16-compressed. Each device compresses FP32 -> BF16 in a local outbox, cross-device memcpys to the peer's inbox (HALF the PCIe bytes), then decompresses + adds into the local FP32 partial. 6 SYCL submissions/call but PCIe bytes halved -- wins for any tensor where PCIe dominates kernel time. Threshold and BF16 path pattern mirror the CUDA NCCL allreduce. Storage: ONE persistent uint8_t buffer per device, 4 * nelem bytes (matches both path layouts: FP32 nelem floats; BF16 outbox+inbox = 2 * nelem uint16_t each). Single alloc+free per device keeps the SYCL pool's strict-LIFO invariant trivial. Initial impl handles N=2 FP32 contiguous tensors. Other cases return false, causing the meta-backend to use its generic butterfly fallback. Per-call sync is intentionally omitted. SYCL in-order queue semantics ensure that the meta-backend's next compute on the same per-device queue waits for our final ADD, and the next allreduce's first op on the same persistent buffer waits via the same queue. Only comm_free does an explicit final wait. OneCCL is NOT used: OneCCL 2021.17 hardcodes single-device-per-process in communicator_impl.hpp:47 (condition devices.size() == 1), which is incompatible with llama.cpp's single-process multi-GPU model. Measured on dual Intel Arc Pro B70 (NEO 26.05.x, oneAPI 2025.3 + DPC++ nightly): Llama-3.3-70B Q4_K_M, -sm tensor -fa 1 -ctk f16 -ctv f16: pp512 = 377.08 t/s (vs 313.65 layer mode = +20.2%) tg128 = 17.40 t/s (vs 9.74 layer mode = +78.6%) Qwen3-Coder-Next-80B-A3B Q3_K_M (MoE): pp512 = 216.56 t/s (vs 156.58 meta-backend butterfly = +38.3%) tg128 = 17.60 t/s (vs 14.31 meta-backend butterfly = +23.0%) Qwen3-4B Q4_K_M: pp64 = 984.51 t/s, tg16 = 49.29 t/s Llama-3.3-70B in SYCL TP now comfortably beats production layer mode on both prefill and decode. Coder-Next-80B-A3B (MoE) also wins on both — the BF16 path is what unlocks the many-medium-allreduces prefill pattern. Build/CMake: no changes. No new dependencies. ~210 lines added across ggml-sycl.h and ggml-sycl.cpp. * Fix comments * documentation update to address PR feedback * Bring over my device-to-device memcpy chagnes * move the dev2dev_memcpy calls to the upstream 7-parameter variety * Fix a typo and remove a trailing whitespace
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Port of Google's TurboQuant (arXiv 2504.19874): WHT-rotated polar quantization for the KV cache, opt-in via
--cache-type-k/-v turbo2|turbo3|turbo4. Core scope — no InnerQ, no head-dim padding (needs 64/128-aligned heads), reference's policy extensions (auto-asymmetric GQA, Boundary-V, layer-adaptive) excluded.Verified per backend — all three tiers, real hardware:
test-quantize-fnspasses (turbo tiers documented-skipped from roundtrip RMSE: they quantize in the rotated domain by design; TQ3_1S passes real thresholds)Portability hardening from the CI matrix (all conventional
fix:commits): RPCGGML_OP_COUNTpin + proto patch bump; C linkage + dllexport for the WHT group-size global (Windows DLL boundary); HIP/MUSA vendor mappings (cudaMemcpyToSymbol,__ballot_sync, explicit__shfl*_syncwidths) and explicit template-instance lists; SYCL: removed 21 unreferenced turbo instances; fixed a latent buffer-overflow in upstream'stest-quantize-fnsscratch sizing (exposed by F32 vec_dot_type, 8KB heap overflow — candidate upstream report); 3 turbo HIP kernels added to the VGPR ignore list per the script's own pattern (259 vs 256 on gfx908).Reading the CI board on this fork:
ubuntu-22-hip-quality-checkstays red inherited from base: upstream master's own runs fail it at our base commit (2 upstream kernels over the VGPR gate); the turbo branch adds zero new offendersKnown gaps: fattn-mma-turbo instances are stubs (the reference ships no MMA implementation anywhere — fused tensor-core turbo prefill is future original work); TQ weight formats compile but are unfocused; Metal set-rows native dispatch pending.