feat(pytorch): add glm5.3-flash support - #4968
qescccczmr wants to merge 7 commits into
Conversation
Reuse MLA/MoE loaders, FA3, mHC, sparse Top-K and compact DeepGEMM. Add KDA state adaptation and multimodal GLM configuration/processing. Preserve existing defaults and the positional MoE prefix argument.
| device=device, | ||
| is_tp=True, | ||
| quant_config=None, | ||
| dp_disable_tp=True, |
There was a problem hiding this comment.
Keep KV-B sharding consistent with the attention TP group
When dp > 1 and attn_tp > 1, dp_disable_tp=True makes kv_b_proj retain all attention heads, while DeepseekV2BMM still shards kc/vc by attn_tp. The inherited process_weights_after_loading() then copies the full KV-B-derived weights directly into these sharded tensors.
|
@qescccczmr Hi, is this PR ready to review? |
| from .step_metadata import register_step_metadata_impl | ||
|
|
||
|
|
||
| def _select_state(state: torch.Tensor, metadata: Any) -> torch.Tensor: |
There was a problem hiding this comment.
can we use existing _state_select , _state_scatter in here?
| head_dim: int, | ||
| lower_bound: float, | ||
| ) -> torch.Tensor: | ||
| if (metadata.spec_state_offsets is not None |
There was a problem hiding this comment.
mtp may need support as well
| inputs = [x.unflatten(1, (batch_size, steps)) | ||
| for x in (mixed_qkv, raw_gate, raw_beta)] | ||
| outputs = [] | ||
| for step in range(steps): |
There was a problem hiding this comment.
this may lead to poor performance. If fla does not provide this verification kernel, we may need to change the tilelang kernel of gated_delta_rule.
3 participants
Summary
Add GLM-5.3 Flash text/image/video integration to the PyTorch engine, prioritizing
existing LMDeploy components over model-specific kernel copies.
and multimodal frontend.
HcPrePost, sparse-index Top-K, and the existing Triton blocked-FP8 MoE pipeline.plus KPool request-tail/cache semantics and attributed BF16 sparse MLA support.
Validation
BuildSpec,KPool to collected/bound cache requests, and KDA metadata to the shared FLA
step updater. Reuse the upstream KV-B post-load weight processing instead of
retaining the old GLM loading workaround. Preserve upstream Ray device setup.
git diff --check: passed.499 passed, 12 skipped, 1 deselected; engine/input/health/Mooncake regressions: 176 passed, 1 skipped.
the same
No CUDA GPUs are availablefailure was reproduced on clean base2928f477with GPUs hidden. No unrelated fix is bundled.Accuracy
These are completed pre-cleanup/pre-merge integration runs, not a fresh benchmark
of the final commit. Local deployment: GLM-5.3 Flash FP8 on H200, TP4/EP1 with Ray and
BF16 KV cache.
MMMU-Pro details
by +0.06 and +0.64 percentage points, respectively.
NVIDIA model card.
Reproduction and limits
Checkpoint revision:
3f1971b7b5f7a528c9c4ef6212c8785298a8c24a.MMMU-Pro runtime: session length 344064, max batch 8, cache fraction 0.4,
cache block 64, prefill chunk 2048, CUDA Graph buckets 1/2/4/8, client concurrency 4.
MTP and latest upstream synchronization (2026-09-21)
9adf4856by normal merge; implementation commit6218f599.deepseek_mtp, the shared GLM predictor, embedding/head, rejection sampler,cache planner, FLA kernels and CUDA Graph mixins. Six existing production files
changed; no new production files or kernels for MTP.
The single predictor layer reconstructs its KPool tail from pageable token data.
Multimodal embeddings use the existing target-to-draft handoff.
and unused EP1 extension remain removed. KPool still uses DeepGEMM scoring.
issue; unrelated models retain their existing behavior.