fix(update_weight): restore FlashInfer MoE layout after BF16 hot updates#2192
Open
LLMShark wants to merge 1 commit into
Open
fix(update_weight): restore FlashInfer MoE layout after BF16 hot updates#2192LLMShark wants to merge 1 commit into
LLMShark wants to merge 1 commit into
Conversation
f453551 to
0a9eeff
Compare
0a9eeff to
0af60fc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
When performing BF16 weight hot-updates with the
flashinfer_trtllmMoE backend, such as Qwen3.5 / Qwen3.6 35B-A3B, SGLang restores block-layout weights to canonical layout before copying the updated weights.However, slime previously called:
only for the
compressed-tensorsquantization path. The BF16 path skipped this step, leaving MoE weights in canonical layout after the update while the FlashInfer TRT-LLM forward path expected block layout. The next forward pass could therefore fail with:at
flashinfer::Bf16MoeLauncher::check_moe().Changes
Updated both weight-update implementations:
slime/backends/megatron_utils/update_weight/update_weight_from_tensor.pyslime/backends/megatron_utils/update_weight/update_weight_from_distributed.pyThe post-load
post_process_weights(...)call now runs after every successful hot-update, rather than only forcompressed-tensors.Additional behavior preserved:
compressed-tensorspre-load call withrestore_weights_before_load=Trueremains conditional.continue_generationis called only after post-processing succeeds.Relationship to SGLang #28015
This PR and SGLang #28015 are independently mergeable and address
different phases of the weight-update lifecycle:
flashinfer_trtllmBF16 MoE backend.canonical weights have been copied.
For the non-routed
flashinfer_trtllmbackend, both changes (orequivalent downstream patches) are required for an end-to-end hot
update:
For routed backends whose pre-load restore path already works, this PR
can independently fix the missing post-load finalize step.
Tests
Added
tests/test_bf16_weight_post_process.py, covering:restore_weights_before_load=Falsepost_process_quantization=Truecompressed-tensorspath preserves the restore-then-post-process order.continue_generationis not called when post-processing fails.The test is included in the CPU CI matrix.
Validation
git diff --checkpassed.test_bf16_weight_post_process.py.colocated setup with the corresponding SGLang patch:
(132/132 and 136/136 update chunks).
ref_log_probs,and
actor_train.reproduced.
This validation confirms the update/finalize control flow and kernel
execution. It does not by itself establish logit or log-probability
parity between the training and rollout engines. Numerical parity
should be validated separately with a fixed-token, no-op hot-update
test.