English · 简体中文
🛠️ Upstream work · 🐋 mini-DeepInfer · 📚 Engineering notes
LLM inference systems & Ascend NPU operators
I work on SGLang and sgl-kernel-npu, connecting serving-runtime behavior with the operators underneath it: MoE expert migration, speculative-decoding state, quantized attention, and numerical correctness.
My contributions span implementation, bug diagnosis, reproducible checks, and technical documentation for DeepSeek-V4 / V4.1 on Ascend.
Python PyTorch / torch_npu CANNBot-DSL HCCL / DeepEP
sgl-kernel-npu #964 ✅ Merged · 2026-11-09 · Co-developed with ClownBin
Integrated seven CANNBot-DSL operator APIs—four compute operators and three metadata builders—for MXFP4 K-cache updates, two-stage sparse indexing, and mixed FP8/FP4 MLA.
My integration work covers API and dispatch contracts, independent Torch references, tests, bilingual API docs, and a backport that pads K-projection backing storage for tail tiles. Contribution commits
SGLang #47574 ✅ Merged · 2026-11-02
Implemented dynamic expert load balancing on Ascend, connecting valid-route statistics, expert-weight migration, and bounded temporary memory.
Bug discovery → merged community fix: I identified and reproduced empty draft records consuming the target model's EPLB history (issue #48603) and supplied the recorder-lifecycle fix in #47922; the community integrated that core fix in #48751, authored by ch-wan and merged on 2026-11-06.
🔬 What made this bug subtle?
Draft execution can use TARGET_VERIFY, so the guard must follow the runner's role. Disabling counting hooks inside the model still lets the outer recorder reset counters and append an empty observation.
The fix has two parts: disable recording before entering the forward context, and make an already-disabled recorder skip reset, collection, and append. The issue includes a current-main reproducer and published 16-rank service records.
Open follow-up: #48835 · Preserve arch35 expert storage during migration.
| Area | Engineering change | PR |
|---|---|---|
| PD state transfer | Peer-local C4 rows and empty state components across PP stages | #47934 / #47937 / #47938 |
| Decode memory | Bound DeepSeek-V4 SWA storage by request slots | #47740 |
| Numerical precision | Preserve Softplus accuracy in GDN gating and decode | kernel #933 |
| Sparse attention | Exclude padded lanes from non-power-of-two TopK reduction | kernel #935 |
| Quantization | Preserve AWQ fallback dtype and correct GPTQ zero-point offsets | #47639 / #47637 |
| Benchmark timing | Record PD decode handoff time for non-streaming requests | #47772 |
Selected patches and the arch35 follow-up were Open when checked on 2026-11-09. Follow each PR for its latest status.
📦 More sglang contributions (12 PRs)
- #47721 · Bound DeepSeek-V4 RoPE tables by serving context · Open
- #47720 · Bound DeepSeek-V4 prefill query intermediates · Open
- #47718 · Release temporary DeepSeek-V4 RoPE tables · Open
- #47713 · dsv4: Reuse verify sequence lengths · Open
- #47665 · dsv4: Generate compression positions with stepped arange · Open
- #47663 · dsv4: Reuse CPU query lengths for prefill metadata · Open
- #47659 · MLA: preserve NZ cache data during CPU retraction · Open
- #47656 · moe: Honor SwiGLU clamp for BF16 and INT8 experts · Open
- #47654 · sparse-kv: Preserve shared prefixes in physical host slots · Open
- #47647 · dsv4: Avoid unused packed indexer cache allocation · Open
- #47634 · attention: Slice MLA latent K after TP all-gather · Open
- #47629 · moe: honor simple TopK scoring and routing scale · Open
📦 More sgl-kernel-npu contributions (8 PRs)
- #936 · include dtypes in legacy add RMSNorm cache key · Open
- #934 · compute weightless RMSNorm in float32 · Open
- #932 · normalize strided gating inputs · Open
- #930 · preserve tiling configurations across graph replays · Open
- #926 · include sparse attributes in the tiling cache key · Open
- #925 · handle cache-location windows larger than 16 tokens · Open
- #924 · handle fully masked leading tiles in argmax probability · Open
- #922 · preserve arch22 accumulation order across batch sizes · Open
A runnable teaching inference engine built around DeepInfer's execution design: scheduling, batching, KV-slot reclamation, execution across iterations, and speculative decoding. Start on CPU, then generate text with real Qwen3.5 weights on Ascend 910C.
Chinese engineering articles connecting SGLang source code with DeepSeek Attention / MoE / DSpark, distributed state ownership, quantization, profiling, and measured tuning results.
Always happy to talk inference, kernels, and a well-reproduced bug 🐋
DeepSeek whale girl · Community fan art