feat(gguf): apply the weight scales of nvfp4 gguf tensors

[jason-fxz]

Part of #671.

Summary

llama.cpp's converter turns a ModelOpt NVFP4 checkpoint into GGUF by repacking each NVFP4 tensor and writing its second-level scale (weight_scale_2) as a separate fp32 tensor <name>.scale: one value for a dense tensor, one per expert for ffn_{gate,up,down}_exps. llama.cpp multiplies every NVFP4 matmul result by it. FreeToken did not read these tensors: Gemma-4 26B-A4B NVFP4 GGUF loaded but generated only <pad>, the Qwen and Llama loaders refused such files, and the qwen4exp loader loaded them without the scales.

This PR applies the scales on every path that computes NVFP4 GGUF weights (GPU dense linears, GPU MoE experts with fused, offload and hybrid, CPU MoE experts with hybrid and cpu) and lets the gemma4, llama, qwen3, qwen3moe, qwen35 and qwen35moe loaders read them. On real NVFP4 GGUFs, greedy decode matches FreeToken's HF NVFP4 path token for token on Gemma-4 26B-A4B and Qwen3.6-35B-A3B, and on both offload, hybrid and cpu give the same tokens as fused.

Design

The scale multiplies a matmul result, so it goes where llama.cpp puts it (build_lora_mm, build_moe_ffn):

dense:   y = (x @ W^T) * s                                       s per tensor
experts: h = x @ W_gate_up^T;  g = h_gate * s_gate,  u = h_up * s_up
         y = (glu(g, u) @ W_down^T) * s_down * routing_weight    s per expert

• Dense linears: a fused module (qkv, gate_up) runs several tensors as one matmul, so the loader stores ggml_scale, one fp32 value per output row (each part's scale, 1 for parts without one), and the linear multiplies its result by it.
• Experts: each MoE layer gets a global bank, [E, 4] fp32 = (gate, up, down, 1), next to gate_up and down. It is copied into the slot with the expert, so offload and hybrid read it like the weights. The vector kernel applies the three scales in its fp32 epilogue (the vendored ggml MMVQ now reads strided scale vectors, i.e. columns of the bank); the matrix kernel path multiplies afterwards.
• CPU experts: a new gguf_scaled CPU MoE format is the GGUF format of feat(cpu-moe): serve GGUF experts on the cpu and hybrid backends through ggml-cpu #704 plus the global bank: gate and up rows are scaled before the GLU, down rows before the routing weight.
• Gemma-4: the converter folds the router's per_expert_scale into the down experts' .scale, so the router uses ones when the down experts carry a scale.
• Loader check: check_scales() accepts a .scale or .input_scale only next to an NVFP4 weight (.input_scale stays unused, as in llama.cpp) and refuses any other scale tensor. gemma4, llama, qwen3, qwen3_moe and qwen3_5_moe use it; qwen4exp now refuses every .scale; qwen2, minimax-m2, minimax-m3 and muse-glimmer keep refusing them.
• Refused combinations: --dtype float16 with a file whose NVFP4 tensors carry scales (results are stored in the activation dtype before the scale, which is about 1e-4, multiplies them, so fp16 overflows), as for MXFP8; per-expert scales on a merged ffn_gate_up_exps NVFP4 tensor (llama.cpp drops them too).

Tested on

NVIDIA H100 80GB HBM3, Intel Xeon Platinum 8480+, driver 580.95.05, CUDA 13.0, torch 2.14.1+cu130, Triton 3.8.0. main is 11c1adb, the base of this branch.

GGUFs: nvidia/Gemma-4-26B-A4B-NVFP4-GGUF as published; the others converted from the ModelOpt NVFP4 checkpoints below with llama.cpp e937b17a2, non-NVFP4 tensors kept in BF16:

python convert_hf_to_gguf.py <hf_dir> --outfile <model>.gguf --outtype bf16

Results

Greedy decode of AIME25 problem 1 through the chat template, 256 tokens (LLM.generate, temperature 0, CUDA graphs). Each cell is the number of tokens identical to the reference before the first difference, out of 256. The llama.cpp reference is llama-server f3363593f on the same GGUF (-ngl 999, /completion with the same prompt token ids, temperature 0, top-k 1). hybrid runs a slot cache of twice the expert count with --moe-hybrid-max-fetch 0, so most routes go to the CPU.

Model (arch), NVFP4 tensors	vs FreeToken HF NVFP4 fused	vs llama.cpp	offload / hybrid / cpu vs fused
Gemma-4 26B-A4B (gemma4), experts	256	249	256 / 256 / 256
Gemma-4 31B (gemma4), MLP	248	212	-
Qwen3.6-35B-A3B (qwen35moe), experts, shared experts, output	256	123	256 / 256 / 256
Qwen3.6-27B (qwen35), MLP, output	156	64	-
Qwen3-30B-A3B-Instruct-2507 (qwen3moe), experts, attention	-	170	256 / 171 / 171
Qwen3-32B (qwen3), attention, MLP	-	256	-
Llama-3.1-8B-Instruct (llama), attention, MLP	-	11	-

• Where the outputs part, both continue the same derivation in other words. At the Llama split, llama.cpp's top two log-probabilities are -1.0404 and -1.0414.
• FreeToken's HF loader fails on the last three checkpoints (chunk expects at least a 1-dimensional tensor), so they are compared with llama.cpp only.

Other checks

• CPU MoE executor with gguf_scaled on real NVFP4 experts (Gemma-4 26B-A4B layer 3, 16 experts, gelu_tanh and silu, batch 1, 3 and 8) against an fp32 reference built from the dequantized rows and the scales: max |error| / max |reference| is 4.5e-3 to 1.0e-2. The gguf format, which reads no scales, gives NaN on the same rows.
• ggml-org/gemma-4-26B-A4B-it-MXFP4-GGUF: fused and hybrid outputs are identical to main, 256 of 256 tokens.
• check_scales() accepts the local gemma4 MXFP4 and BF16 GGUFs and the Qwen3.8-Flash-Next IQ3_XXS GGUF, which carry no NVFP4 scales. --dtype float16 with Gemma-4 26B-A4B NVFP4 GGUF is refused at startup.
• PYTHONPATH=python pytest tests/kernels/test_ggml.py tests/models tests/moe/test_cpu_moe.py tests/moe/test_mixed_expert_formats.py tests/moe/test_offload.py tests/server/test_parser_auto_selection.py tests/layers -m "not slow" --deselect tests/moe/test_offload.py::test_graph_capture_reuses_warm_offload_cache_before_capture: 477 passed, 179 skipped, 1 failed (the apodex/Apodex-1.1-mini-NVFP4 case of test_quant_config.py). The deselected test and that case also fail on main.

Not covered

• These loaders still refuse NVFP4 .scale tensors:
  • qwen2: no ModelOpt NVFP4 text checkpoint was found.
  • minimax-m3: the checkpoint is 250 GB and was not converted.
  • minimax-m2: llama.cpp's converter fails on MiniMax-M2.5-NVFP4 (Can not map tensor 'model.layers.0.block_sparse_moe.experts.0.w1.weight'); its NVFP4 expert path expects gate_proj / up_proj / down_proj names.
  • muse-glimmer: the converter's NVFP4 path skips the q/k norms it writes for BF16 attention and the q/k rope unpermute, and llama.cpp requires attn_q_norm, so such a file does not load in llama.cpp either.
  • qwen4exp: not tested; the only NVFP4 checkpoint here is 222 GB.
• NVFP4 token_embd is still refused: the vendored get_rows has no NVFP4 case.
• Dense NVFP4 linears multiply by the scale in a separate elementwise kernel after the ggml GEMM.
• With FREETOKEN_FUSED_COPY=0, a scaled GGUF fails at startup: the per-bank copy needs each bank's slot size to be a multiple of 128 bytes, and a global slot is 16 bytes. With --moe-prefill-hit-d2d, global is copied as one small batch entry.
• Only H100 was tested. ROCm was not run.