feat(models): port Qwen3-Omni MoE thinker (Qwen3-VL-MoE + audio tower, stage 1)#664
Merged
Conversation
…, stage 1)
Add the Qwen3-Omni MoE thinker (qwen3_omni_moe): text output conditioned on text, image, and audio inputs. The vision tower and MoE text decoder reuse the in-tree Qwen3-VL-MoE stack unchanged (DeepStack injection, interleaved MRoPE, per-sequence MRoPE state); the new audio tower (src/audio/qwen3_omni_moe/) converts 16 kHz audio to a 128-bin log-mel spectrogram (whisper_mel), splits it into 100-frame chunks, downsamples through three stride-2 conv + GELU stages (13 output frames per second, out_len(L) = 13*(L/100) + convdown3(L%100), unit-tested), adds a construction-time sinusoidal position table that restarts per chunk, gathers per-chunk valid frames, and runs 32 pre-LN encoder layers with block-diagonal attention over 104-frame windows before projecting to the decoder width. Audio rows scatter at audio_token_id positions like image features; audio placeholders keep sequential text positions in the MRoPE index so compute_rope_index needs no audio awareness. The loader parses the nested thinker_config (stripping nulls and the in_channels/in_chans alias pair), remaps both converted (thinker.language_model.*, thinker.vision_tower.*) and raw (thinker.model.*, thinker.visual.*) key spellings, and drops the talker.*/code2wav.* speech stack before remap. The runtime gives the model a dedicated arm whose no-placeholder image splice lands at the end of the last user turn; the CLI --audio path and the server input_audio path (expander + prepare fn mirroring the Nemotron omni pattern) are both wired, combinable with images. Also fixes CLI/server chat-template parity: the CLI now defaults enable_thinking=true when the tokenizer has a recognized think-marker pair (upstream mlx-lm PR #1114 semantics, matching server startup); without this, Qwen3-family templates rendered an empty think block that the non-thinking omni thinker answered with an immediate end-of-text. Validated against mlx-community/Qwen3-Omni-30B-A3B-Instruct-4bit: text ("The capital of France is Paris."), word-exact audio transcription through both the CLI and the server input_audio path ("The quick brown fox jumps over the lazy dog."), left/right color localization on images through the CLI and server image_url path, and combined image + audio requests. Unit, detection, and real-model parity tests pass; fmt and clippy clean.
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the Qwen3-Omni MoE thinker (
qwen3_omni_moe), stage 1 of Alibaba's omni-modal MoE: text output conditioned on text, image, and audio inputs. The talker / code2wav speech-output stack is out of scope (weights dropped at load; follow-up issue to come).vision_configdeserializes intoQwen3VLVisionConfigdirectly.src/audio/qwen3_omni_moe/): 16 kHz audio to 128-bin log-mel (reusingwhisper_mel), 100-frame chunking, three stride-2 conv + GELU stages (out_len(L) = 13 * (L/100) + convdown3(L%100), unit-tested), a construction-time sinusoidal position table restarting per chunk (never loaded), per-chunk valid-frame gather, and 32 pre-LN layers with block-diagonal attention over 104-frame (8 s) windows, projected to the decoder width. Audio rows scatter ataudio_token_idpositions; audio placeholders keep sequential text positions in the MRoPE index, socompute_rope_indexneeds no audio awareness.load_qwen3_omni_moe): parses the nestedthinker_config(null-stripping; thein_channels/in_chansalias pair is deduplicated), remaps both converted (thinker.language_model.*,thinker.vision_tower.*) and raw (thinker.model.*,thinker.visual.*) spellings, dropstalker.*/code2wav.*before remap so those arrays are freed, and tolerates their absence.insert_qwen3_omni_image_tokens); CLI--audio(combinable with--image) and the serverinput_audiopath (expander + prepare fn mirroring the Nemotron H Nano Omni pattern) are both wired.Chat-template fix (found by this port)
The CLI rendered Qwen3-family templates with
enable_thinkingalways definedfalse, which emits an empty<think>\n\n</think>block. The non-thinking omni thinker answers that out-of-distribution block with an immediate end-of-text (0 generated tokens). The CLI now mirrors server startup (upstreamml-explore/mlx-lmPR #1114 semantics): a tokenizer with a recognized think-marker pair defaultsenable_thinking=true, so the template'senable_thinking is defined and enable_thinking is falsebranch no longer fires.Validation
Real checkpoint
mlx-community/Qwen3-Omni-30B-A3B-Instruct-4bit(30B-A3B; text 4-bit with pre-stacked experts, vision and audio towers bf16):--audioand serverinput_audioeach return "The quick brown fox jumps over the lazy dog." for a synthesized speech clip (249 mel frames -> 33 audio tokens, matchingout_len).image_urlpath ("the color on the left side is red, and the color on the right side is blue").Unit tests (audio
out_len, detection, config parsing), 3 real-model parity tests plus an ignored greedy-dump diagnostic,cargo fmt, andcargo clippy --all-targets -- -D warningsare clean.Out of scope
Talker / code2wav speech output (
enable_audio_output,tts_*tokens),use_audio_in_videotimestamp interleaving, video inputs, and multi-clip audio batching (first clip wins, with a warning).Closes #534