fix(qwen3.5-moe): sanitize toward the gate/up/down expert names SparseMoeBlock actually loads#671
Merged
Merged
Conversation
…eMoeBlock actually loads PR #588 renamed the shared qwen3_next SwitchGLU/SparseMoeBlock expert projections from w1/w2/w3 to gate_proj/up_proj/down_proj, but qwen3_5::sanitize_weights kept normalizing in the OLD direction: step 6 stacked per-expert experts.{e}.{w1,w2,w3} into switch_mlp.{w1,w2,w3}, and step 8 renamed pre-stacked switch_mlp.{gate,up,down}_proj INTO switch_mlp.{w1,w3,w2}. Every qwen3.5-MoE / qwen3.6-MoE checkpoint has therefore failed to load since #588 with "Missing weight: model.layers.0.mlp.switch_mlp.gate_proj" (issue #670; found by the 2026-07-06 benchmark sweep, where qwen3.5-35b-a3b-4bit and qwen3.6-35b-a3b-4bit regressed from 70/67 tok/s on 2026-06-12 to FAIL). The #587 per-expert gate/up/down stacking pass also targeted the stale w1/w3/w2 slots; its tests asserted sanitize output keys only and never loaded the result, so they passed against the wrong convention. The fix aligns all three passes with the block naming: per-expert w1/w2/w3 stacks into gate_proj/down_proj/up_proj, per-expert gate/up/down stacks into the same-name slots, and step 8 now renames legacy stacked w1/w3/w2 forward into gate/up/down while pre-stacked gate/up/down checkpoints pass through untouched. The missing gate is added as a regression test: sanitize output must load end to end through the shared SwitchGLU for all four naming conventions checkpoints ship (per-expert and pre-stacked, in both namings). Real-checkpoint validation: qwen3.5-35b-a3b-4bit and qwen3.6-35b-a3b-4bit load and answer coherently again (57.8 and 72.8 tok/s; "The capital of France is Paris." at a 200-token budget through the thinking template), and dense qwen3.5-4b-4bit is unaffected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Every qwen3.5-MoE / qwen3.6-MoE checkpoint has failed to load since PR #588 with
Missing weight: model.layers.0.mlp.switch_mlp.gate_proj. Found by the 2026-07-06 full benchmark sweep:qwen3.5-35b-a3b-4bitandqwen3.6-35b-a3b-4bitbenched at 70 / 67 tok/s on 2026-06-12 and wereFAIL:benchon 2026-07-06.Root cause
PR #588 renamed the shared
qwen3_nextSwitchGLU/SparseMoeBlockexpert projections fromw1/w2/w3togate_proj/up_proj/down_proj, butqwen3_5::sanitize_weightskept normalizing in the OLD direction: step 6 stacked per-expertexperts.{e}.{w1,w2,w3}intoswitch_mlp.{w1,w2,w3}, and step 8 renamed pre-stackedswitch_mlp.{gate,up,down}_projINTOswitch_mlp.{w1,w3,w2}. The block then requestsswitch_mlp.gate_projand finds nothing. PR #587's per-expert gate/up/down stacking pass also targeted the stalew1/w3/w2slots; its tests asserted sanitize OUTPUT keys only and never loaded the result, so they passed against the wrong convention — the same "synthetic passes, real checkpoint fails" class as #588 root-caused.Fix
All three passes now align with the block naming: per-expert
w1/w2/w3stacks intogate_proj/down_proj/up_proj(w1→gate, w2→down, w3→up), per-expertgate/up/downstacks into the same-name slots, and step 8 renames legacy stackedw1/w3/w2FORWARD intogate/up/downwhile pre-stacked gate/up/down checkpoints pass through untouched.Validation
SwitchGLUfor all four naming conventions checkpoints ship (per-expert and pre-stacked, in both namings). It fails on the pre-fix sanitize by construction (the old direction producesw1keys the block never requests).qwen3.5-35b-a3b-4bitandqwen3.6-35b-a3b-4bitload and generate again (57.8 / 72.8 tok/s; "The capital of France is Paris." at a 200-token budget through the thinking template); denseqwen3.5-4b-4bitunaffected.cargo fmt,cargo clippy --all-targets -- -D warnings, and the qwen3_5 unit suite are clean. The two models will be re-added to the 2026-07-06 benchmark CSV after merge.Closes #670