Repository navigation
feat(gguf): serve GLM-5.3-Flash GGUF checkpoints on glm5-next - #721
Conversation
492e79f to
b463832
Compare
Test stats & setupWorkload results (bs=1, single RTX 5090):
Model: unsloth Launch: Host: Ryzen 9 7950X3D (32 threads), 187 GiB RAM (~130 GiB pinned expert banks), RTX 5090 32 GB (sm_120), driver 580.95.05, CUDA 13.0, Linux 6.8.0-136-generic, shards on a Crucial T500 2 TB NVMe. Stack: Python 3.10.19, torch 2.14.1, triton 3.8, transformers 5.18.0. @jason-fxz Upstreaming my private branch that I was using for GLM 5.3 Flash GGUF. Rebased to current main to align this branch with the gguf direction chosen by the Repo. I also have a branch up for deepseek v4 privately, if you'd like me to help you finish these gguf ports, let me know :) |
|
Could you first rebase onto the current main? The branch conflicts with it, and these GGUF changes landed since its base:
Then, before this goes further, could you run the GGUF load check on the GLM-5.3-Flash files? It is on the git fetch https://github.com/FlashML-org/FreeToken feat/gguf-check-tool
git checkout FETCH_HEAD -- scripts/gguf_check
# header-only stubs of every GLM-5.3-Flash GGUF on HF (about 15 MB each)
PYTHONPATH=python python scripts/gguf_check/fetch_headers.py ~/gguf-stubs --search GLM-5.3-Flash > fetch.jsonl
# kernel table, once per GPU architecture
PYTHONPATH=python python scripts/gguf_check/probe_kernels.py kernels.json
# CPU only, no weights read
PYTHONPATH=python python scripts/gguf_check/check.py --kernels kernels.json --out glm53.jsonl ~/gguf-stubsFor each file it checks that the config parses, that the model builds and the reader yields exactly its state dict, that the expert banks build with the right shapes, that every ggml type has its kernels, and that the tokenizer and chat template load. The bar: the common quants from unsloth, ggml-org, bartowski and others should pass. Every failure should be a clear refusal from the loader or a broken file. Please post the These failures are things we do not serve today, so you can ignore them:
For NVFP4 load: the |
2ef2880 to
9649143
Compare
check_scales at config parse like the llama, qwen3 and gemma4 loaders (review on FlashML-org#721): a .scale / .input_scale outside an NVFP4 weight now fails with a clear error instead of loading silently unread.
|
Hi @jason-fxz , thanks for the review. I've Rebased, Rebase & #732
GGUF load checkI could not check every GLM-5.3-Flash file on HF, so I ran it on the usual, most-downloaded ones:
Failures, grouped: 21 mixed gate/up expert types, 3 missing Not covered
Fixes the check + serving run caught (on the branch, each with a regression test)
Test stats & setupbs=1, single RTX 5090, on unsloth
Tuned launch (400k context, bigger prefill chunks, more expert slots): Host: Ryzen 9 7950X3D, 187 GiB RAM (~134 GiB pinned expert banks), RTX 5090 32 GB (sm_120), |
|
Follow-up: I let the check run against every GLM-5.3-Flash GGUF HF search finds, -- 2,118 shard files across 107 repos, grouped into split sets, same kernel table: The long tail found one more writer-variant pair, fixed on the branch with regression tests:
Failures, grouped (full sweep)
Nothing crashes and nothing mis-loads silently. Not covered (additions to the list above)
|
|
Thanks for the rebase and the fixes. I re-checked at 9649143, also merged onto current main (bb15d1d, clean merge):
Nothing blocking. A few things to clean up before merge: 1. Which public GGUF checkpoint needs the metadata fallbacks? The tensor-name census in 2. Refuse metadata the model can't express instead of reading part of it. llama.cpp reads these per layer (
Every published file has uniform values here (clamp 10 / 10, sigmoid, all-full indexer), so this changes no current file. It turns a future mismatch into an error instead of wrong output. 3. Tests. Please remove 4. Description and commit message. The PR body and the first commit message still say this PR adds the Nits:
|
Adds the llama.cpp glm5-next GGUF reader (unsloth GLM-5.3-Flash-GGUF, both the glm5-next and the older glm5next arch spellings) on the GGUF quant kind from FlashML-org#690: - parse_gguf_config derives the text config from the GGUF KV metadata, with per-block attention.head_count_kv laying out the KDA/DSA hybrid, the trailing MTP block dropped like the HF loader drops it, and the MLA head widths from the *_length_mla keys. - iter_gguf_weights maps every tensor to the glm5_next state dict and undoes the converter transforms: A_log stored as -exp(A_log) is inverted, the per-head attn_k_b/attn_v_b pair (k transposed) fuses into the dense kv_b_proj layout the MLA absorption reads, and the q|k|v conv kernels concatenate on the channel axis. The mixed-type fused KDA in_proj (Q8_0 q|k|v + BF16 b|f_a|g_a) rides FlashML-org#702 segments; the routed experts load as GGUF expert banks at their shipped types (unsloth dynamic quants: IQ3_S gate/up + IQ4_XS down, Q8_0 outlier layers pass through natively). - passes quant_config to embed_tokens so the Q8_0 token_embd serves packed; registers Glm5NextGGUFForCausalLM and claims it in the AOT table. Verified end to end against unsloth GLM-5.3-Flash-UD-IQ4_XS (5 split shards): the full engine on a 32GB GPU serves coherent output with the experts off host banks. Text-only, TP=1. Part of FlashML-org#671.
check_scales at config parse like the llama, qwen3 and gemma4 loaders (review on FlashML-org#721): a .scale / .input_scale outside an NVFP4 weight now fails with a clear error instead of loading silently unread.
…uants unsloth's UD quants quantize tensors the AesSedAI-style files keep float: the MLA kv_b halves (Q8_0), the mHC mixers and the indexer's wk / weights_proj / compressor gate. kv_b_proj joins the registry's unquantized_modules (glm_moe_dsa keeps its kv_b dense for the MLA absorption for the same reason) and the reader dequantizes through the vendored gguf-py reference on every float-expecting read; wq_b keeps serving its checkpoint quant packed (attention.py keeps wk and weights_proj dense). Regression test stores the kv_b halves Q8_0.
llama-vocab.cpp maps both spellings to CHATGLM4 with ignore_merges; unsloth writes glm4 and AesSedAI's writer glm5 for the same vocabulary. Without the mapping a glm5 file loads but silently mis-tokenizes (tok_compare: 22,840/153,725 vocab words differ from the HF reference; 0 everywhere with it).
pad_rows added one zero row of k columns, which covers the bytes MMQ reads past the last row only when k is at least half a padding unit (512); glm5-next's KDA f_b / g_b (k = linear head dim 128, Q8_0 in dynamic quants) are the first packed tensors below that and died at the first mul_mat. pad_rows now takes the ggml type and returns enough whole rows to cover mmq_tail_bytes for any (type, k), consistently in the reader's fused(), the gguf linear and embedding params and the expert host banks. Found serving the real unsloth UD-IQ4_XS: the dry load never runs a forward.
Two long-tail writer variants the full check sweep found (the GSQ-RCO lineage alone is ~57k downloads/month): - the conv kernels with a trailing unit dim: llama.cpp's glm5-next loader tries that 4-D shape before the 3-D one, so trailing unit dims in ne are dropped on float reads (they carry no elements). - attention.head_count_kv as the generic scalar (all blocks): wrong for the hybrid -- llama.cpp broadcasts it and then fails tensor creation -- so only the list is trusted and a scalar falls to the tensor-name census like the pre-list writer.
- refuse instead of guessing metadata the model cannot express: the scalar attention.head_count_kv spelling (llama.cpp broadcasts it and then fails tensor creation; every public file carries the list), missing *_length_mla keys through need(), non-uniform swiglu_clamp_exp / swiglu_clamp_shexp, indexer.types with non-full layers, expert_gating_func other than sigmoid, and rope dims as a NotImplementedError instead of an assert. - call check_scales directly; aot_models comment on its own line. - drop the GGUF test files; coverage will be handled across families.
|
Sorry! I think I pushed a commit right before you checked it. All four fixes applied (plus the nits), force-pushed on top of
Re-ran your spot-check set on the result: unsloth 11/12 (UD-IQ1_M only), ggml-org Q2_K_S, |
|
The branch head is still 7ec6289, and I don't see the changes there (test files are still in). Did the push go through? |
7ec6289 to
b9e4849
Compare
|
I'm so sorry, I didn't press enter 🙃 My bad |
2 participants
Part of #671 (Model Support:
glm5-next).Follows the per-family GGUF adapter pattern established for qwen4exp in #690 (config from KV metadata through the family's own
parse_config, checkpoint-leafto_ggufmap,fused()state-dict emission, GGUF expert banks, text-only TP=1).What
Serves llama.cpp
glm5-nextGGUFs — unsloth GLM-5.3-Flash-GGUF — on the GGUF quant kind from #690, reusing the existingglm5_nextmodel classes:parse_gguf_configbuilds the text config from the GGUF KV metadata. The writer carriesattention.head_count_kvas a per-block list (0 = KDA, 1 = MLA), which encodes the 34/11 hybrid layout directly; the older unslothglm5nextspelling carries it too and stays as an alias. The trailing MTP block (nextn_predict_layers) is dropped exactly like the HF loader drops the MTP layer, and the MLA head widths come from the*_length_mlakeys. Metadata the model cannot express — a scalarhead_count_kv, non-uniformswiglu_clamp_exp/swiglu_clamp_shexp, indexer types without a full indexer,expert_gating_funcother than sigmoid, rotary dims on the main attention — is refused with a clear error, as llama.cpp does.iter_gguf_weightsmaps every tensor into theglm5_nextstate dict and undoes the converter transforms:A_logis stored pre-transformed as-exp(A_log)and inverted at load; the MLAkv_b_projis split per-head asattn_k_b/attn_v_b(k transposed) and re-fused into the dense[H*(nope+v), lora]layout the module's absorption reads, dequantizing when the quant stored it packed; the q|k|v conv kernels concatenate on the channel axis (4-D spellings with a trailing unit dim included, as llama.cpp reads them). The mixed-type fused KDAin_proj(Q8_0 q|k|v + BF16 b|f_a|g_a, as shipped) rides the feat(gguf): run experts whose gate_up and down mix float and quantized types #702 segments.Glm5NextGGUFForCausalLMregistered and claimed in the AOT table;embed_tokenstakesquant_configso the Q8_0token_embdserves packed.check_scales(feat(gguf): apply the weight scales of nvfp4 gguf tensors #732) runs at config parse like the llama, qwen3 and gemma4 loaders.Verified
GLM-5.3-Flash-UD-IQ4_XS(5 split shards, sha256-checked against the repo): the full 45-layer engine (RTX 5090 32GB, 40k context, 6 GiB expert slot cache) loads the experts as host banks and serves coherent, correct output over/v1/chat/completions; the tokenizer, eos set and chat template come from the GGUF via the shared loader.zai-org/GLM-5.3-Flashconfig.json throughglm5_next.config.parse_config; the state dict matches the constructed model 1042/1042 tensors (names + shapes), re-verified byscripts/gguf_check's dry load on the real file.tok_compare: 0 differing texts vs the HF tokenizer on all 153,725 vocab words and 20k random combos.scripts/gguf_checkover every GLM-5.3-Flash GGUF HF search finds (2,118 shards / 107 repos → 784 served-arch files): 267 pass; every failure is on the ignore list, a not-model file (per-layer exports, drafts), or a clear refusal. The common quants from the big repos (unsloth, orcarouter, Maya, huihui-ai, autotrust, ggml-org, GSQ-RCO, AesSedAI) all pass.Text-only, TP=1 — same status as the other GGUF families. Vision (
mmprojsidecar) is a natural follow-up.