Skip to content

feat(models): port Qwen3-Omni MoE thinker (Qwen3-VL-MoE + audio tower, stage 1)#664

Merged
inureyes merged 1 commit into
mainfrom
feat/issue-534-qwen3-omni
Jul 5, 2026
Merged

feat(models): port Qwen3-Omni MoE thinker (Qwen3-VL-MoE + audio tower, stage 1)#664
inureyes merged 1 commit into
mainfrom
feat/issue-534-qwen3-omni

Conversation

@inureyes

@inureyes inureyes commented Jul 5, 2026

Copy link
Copy Markdown
Member

Summary

Adds the Qwen3-Omni MoE thinker (qwen3_omni_moe), stage 1 of Alibaba's omni-modal MoE: text output conditioned on text, image, and audio inputs. The talker / code2wav speech-output stack is out of scope (weights dropped at load; follow-up issue to come).

  • Reused unchanged: the Qwen3-VL-MoE text decoder (interleaved MRoPE, DeepStack, per-sequence MRoPE state) and the Qwen3-VL vision encoder; the omni vision_config deserializes into Qwen3VLVisionConfig directly.
  • New audio tower (src/audio/qwen3_omni_moe/): 16 kHz audio to 128-bin log-mel (reusing whisper_mel), 100-frame chunking, three stride-2 conv + GELU stages (out_len(L) = 13 * (L/100) + convdown3(L%100), unit-tested), a construction-time sinusoidal position table restarting per chunk (never loaded), per-chunk valid-frame gather, and 32 pre-LN layers with block-diagonal attention over 104-frame (8 s) windows, projected to the decoder width. Audio rows scatter at audio_token_id positions; audio placeholders keep sequential text positions in the MRoPE index, so compute_rope_index needs no audio awareness.
  • Loader (load_qwen3_omni_moe): parses the nested thinker_config (null-stripping; the in_channels / in_chans alias pair is deduplicated), remaps both converted (thinker.language_model.*, thinker.vision_tower.*) and raw (thinker.model.*, thinker.visual.*) spellings, drops talker.* / code2wav.* before remap so those arrays are freed, and tolerates their absence.
  • Runtime: a dedicated arm whose no-placeholder image splice lands at the end of the last user turn (insert_qwen3_omni_image_tokens); CLI --audio (combinable with --image) and the server input_audio path (expander + prepare fn mirroring the Nemotron H Nano Omni pattern) are both wired.

Chat-template fix (found by this port)

The CLI rendered Qwen3-family templates with enable_thinking always defined false, which emits an empty <think>\n\n</think> block. The non-thinking omni thinker answers that out-of-distribution block with an immediate end-of-text (0 generated tokens). The CLI now mirrors server startup (upstream ml-explore/mlx-lm PR #1114 semantics): a tokenizer with a recognized think-marker pair defaults enable_thinking=true, so the template's enable_thinking is defined and enable_thinking is false branch no longer fires.

Validation

Real checkpoint mlx-community/Qwen3-Omni-30B-A3B-Instruct-4bit (30B-A3B; text 4-bit with pre-stacked experts, vision and audio towers bf16):

  • Text: "The capital of France is Paris."
  • Audio, word-exact transcription through BOTH paths: CLI --audio and server input_audio each return "The quick brown fox jumps over the lazy dog." for a synthesized speech clip (249 mel frames -> 33 audio tokens, matching out_len).
  • Image: left/right color localization correct through the CLI and the server image_url path ("the color on the left side is red, and the color on the right side is blue").
  • Combined image + audio requests work on both paths.
  • Observed limit: dense-text OCR on rendered screenshots is weak on this 4-bit instruct thinker (degenerate underscores); scene/color/spatial queries are fine. Noted for a potential follow-up comparison against upstream mlx-vlm.

Unit tests (audio out_len, detection, config parsing), 3 real-model parity tests plus an ignored greedy-dump diagnostic, cargo fmt, and cargo clippy --all-targets -- -D warnings are clean.

Out of scope

Talker / code2wav speech output (enable_audio_output, tts_* tokens), use_audio_in_video timestamp interleaving, video inputs, and multi-clip audio batching (first clip wins, with a warning).

Closes #534

…, stage 1)

Add the Qwen3-Omni MoE thinker (qwen3_omni_moe): text output conditioned on text, image, and audio inputs. The vision tower and MoE text decoder reuse the in-tree Qwen3-VL-MoE stack unchanged (DeepStack injection, interleaved MRoPE, per-sequence MRoPE state); the new audio tower (src/audio/qwen3_omni_moe/) converts 16 kHz audio to a 128-bin log-mel spectrogram (whisper_mel), splits it into 100-frame chunks, downsamples through three stride-2 conv + GELU stages (13 output frames per second, out_len(L) = 13*(L/100) + convdown3(L%100), unit-tested), adds a construction-time sinusoidal position table that restarts per chunk, gathers per-chunk valid frames, and runs 32 pre-LN encoder layers with block-diagonal attention over 104-frame windows before projecting to the decoder width. Audio rows scatter at audio_token_id positions like image features; audio placeholders keep sequential text positions in the MRoPE index so compute_rope_index needs no audio awareness. The loader parses the nested thinker_config (stripping nulls and the in_channels/in_chans alias pair), remaps both converted (thinker.language_model.*, thinker.vision_tower.*) and raw (thinker.model.*, thinker.visual.*) key spellings, and drops the talker.*/code2wav.* speech stack before remap. The runtime gives the model a dedicated arm whose no-placeholder image splice lands at the end of the last user turn; the CLI --audio path and the server input_audio path (expander + prepare fn mirroring the Nemotron omni pattern) are both wired, combinable with images. Also fixes CLI/server chat-template parity: the CLI now defaults enable_thinking=true when the tokenizer has a recognized think-marker pair (upstream mlx-lm PR #1114 semantics, matching server startup); without this, Qwen3-family templates rendered an empty think block that the non-thinking omni thinker answered with an immediate end-of-text. Validated against mlx-community/Qwen3-Omni-30B-A3B-Instruct-4bit: text ("The capital of France is Paris."), word-exact audio transcription through both the CLI and the server input_audio path ("The quick brown fox jumps over the lazy dog."), left/right color localization on images through the CLI and server image_url path, and combined image + audio requests. Unit, detection, and real-model parity tests pass; fmt and clippy clean.
@inureyes inureyes added type:enhancement New features, capabilities, or significant additions area:models Model architectures, weights, loading, metadata priority:medium Medium priority labels Jul 5, 2026
@inureyes inureyes merged commit 222d904 into main Jul 5, 2026
5 checks passed
@inureyes inureyes deleted the feat/issue-534-qwen3-omni branch July 5, 2026 16:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:models Model architectures, weights, loading, metadata priority:medium Medium priority type:enhancement New features, capabilities, or significant additions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(models): port Qwen3-Omni MoE (text + vision + audio MoE)

1 participant