Skip to content

fix(qwen3.5-moe): sanitize toward the gate/up/down expert names SparseMoeBlock actually loads#671

Merged
inureyes merged 1 commit into
mainfrom
fix/issue-670-qwen35-moe-sanitize
Jul 6, 2026
Merged

fix(qwen3.5-moe): sanitize toward the gate/up/down expert names SparseMoeBlock actually loads#671
inureyes merged 1 commit into
mainfrom
fix/issue-670-qwen35-moe-sanitize

Conversation

@inureyes

@inureyes inureyes commented Jul 6, 2026

Copy link
Copy Markdown
Member

Summary

Every qwen3.5-MoE / qwen3.6-MoE checkpoint has failed to load since PR #588 with Missing weight: model.layers.0.mlp.switch_mlp.gate_proj. Found by the 2026-07-06 full benchmark sweep: qwen3.5-35b-a3b-4bit and qwen3.6-35b-a3b-4bit benched at 70 / 67 tok/s on 2026-06-12 and were FAIL:bench on 2026-07-06.

Root cause

PR #588 renamed the shared qwen3_next SwitchGLU / SparseMoeBlock expert projections from w1/w2/w3 to gate_proj/up_proj/down_proj, but qwen3_5::sanitize_weights kept normalizing in the OLD direction: step 6 stacked per-expert experts.{e}.{w1,w2,w3} into switch_mlp.{w1,w2,w3}, and step 8 renamed pre-stacked switch_mlp.{gate,up,down}_proj INTO switch_mlp.{w1,w3,w2}. The block then requests switch_mlp.gate_proj and finds nothing. PR #587's per-expert gate/up/down stacking pass also targeted the stale w1/w3/w2 slots; its tests asserted sanitize OUTPUT keys only and never loaded the result, so they passed against the wrong convention — the same "synthetic passes, real checkpoint fails" class as #588 root-caused.

Fix

All three passes now align with the block naming: per-expert w1/w2/w3 stacks into gate_proj/down_proj/up_proj (w1→gate, w2→down, w3→up), per-expert gate/up/down stacks into the same-name slots, and step 8 renames legacy stacked w1/w3/w2 FORWARD into gate/up/down while pre-stacked gate/up/down checkpoints pass through untouched.

Validation

  • The missing gate is now a regression test: sanitize output must load end to end through the shared SwitchGLU for all four naming conventions checkpoints ship (per-expert and pre-stacked, in both namings). It fails on the pre-fix sanitize by construction (the old direction produces w1 keys the block never requests).
  • The three fix(qwen3.5-moe): stack per-expert gate_proj/up_proj/down_proj weight names #587 tests are retargeted at the gate/up/down expectations and pass.
  • Real checkpoints: qwen3.5-35b-a3b-4bit and qwen3.6-35b-a3b-4bit load and generate again (57.8 / 72.8 tok/s; "The capital of France is Paris." at a 200-token budget through the thinking template); dense qwen3.5-4b-4bit unaffected.
  • cargo fmt, cargo clippy --all-targets -- -D warnings, and the qwen3_5 unit suite are clean. The two models will be re-added to the 2026-07-06 benchmark CSV after merge.

Closes #670

…eMoeBlock actually loads

PR #588 renamed the shared qwen3_next SwitchGLU/SparseMoeBlock expert projections from w1/w2/w3 to gate_proj/up_proj/down_proj, but qwen3_5::sanitize_weights kept normalizing in the OLD direction: step 6 stacked per-expert experts.{e}.{w1,w2,w3} into switch_mlp.{w1,w2,w3}, and step 8 renamed pre-stacked switch_mlp.{gate,up,down}_proj INTO switch_mlp.{w1,w3,w2}. Every qwen3.5-MoE / qwen3.6-MoE checkpoint has therefore failed to load since #588 with "Missing weight: model.layers.0.mlp.switch_mlp.gate_proj" (issue #670; found by the 2026-07-06 benchmark sweep, where qwen3.5-35b-a3b-4bit and qwen3.6-35b-a3b-4bit regressed from 70/67 tok/s on 2026-06-12 to FAIL). The #587 per-expert gate/up/down stacking pass also targeted the stale w1/w3/w2 slots; its tests asserted sanitize output keys only and never loaded the result, so they passed against the wrong convention. The fix aligns all three passes with the block naming: per-expert w1/w2/w3 stacks into gate_proj/down_proj/up_proj, per-expert gate/up/down stacks into the same-name slots, and step 8 now renames legacy stacked w1/w3/w2 forward into gate/up/down while pre-stacked gate/up/down checkpoints pass through untouched. The missing gate is added as a regression test: sanitize output must load end to end through the shared SwitchGLU for all four naming conventions checkpoints ship (per-expert and pre-stacked, in both namings). Real-checkpoint validation: qwen3.5-35b-a3b-4bit and qwen3.6-35b-a3b-4bit load and answer coherently again (57.8 and 72.8 tok/s; "The capital of France is Paris." at a 200-token budget through the thinking template), and dense qwen3.5-4b-4bit is unaffected.
@inureyes inureyes added type:bug Bug fixes, error corrections, or issue resolutions area:models Model architectures, weights, loading, metadata priority:high High priority labels Jul 6, 2026
@inureyes inureyes merged commit 33385e7 into main Jul 6, 2026
5 checks passed
@inureyes inureyes deleted the fix/issue-670-qwen35-moe-sanitize branch July 6, 2026 01:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:models Model architectures, weights, loading, metadata priority:high High priority type:bug Bug fixes, error corrections, or issue resolutions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(qwen3.5-moe): sanitize renames stacked experts to w1/w2/w3 but SparseMoeBlock now loads gate_proj/up_proj/down_proj (load regression since #588)

1 participant