Bug: text-only Qwen3.5/Qwen3.6 MoE models are incorrectly auto-detected as VLM

[traderjoe1968]

Bug: text-only Qwen3.5/Qwen3.6 MoE models are auto-detected as VLM and fail without vision_config

Summary

ModelArchitectureProbe currently treats qwen3_5, qwen3.5, and qwen3_5_moe as unconditional vision model types.

This causes text-only Qwen3.5/Qwen3.6 checkpoints that use model_type: "qwen3_5_moe" to be routed through the VLM loader even when config.json contains no vision_config.

The result is a startup failure such as:

[SwiftLM] Auto-detected VLM config (qwen3_5_moe); enabling vision mode.
[SwiftLM] Loading VLM (vision-language model)...
Error: Failed to parse config.json ... Missing field 'vision_config'

Reproduction

Using current main:

.build/release/SwiftLM \
  --model destynova002/Qwen3.6-35B-A3B-mixed-3-4bit-3.85bpw-mlx \
  --port 5413 \
  --ctx-size 14336 \
  --mem-limit 8192 \
  --max-tokens 1536 \
  --parallel 1 \
  --stream-experts \
  --ssd-prefetch

The model is text-only and has no vision_config, but SwiftLM detects qwen3_5_moe as VLM.

Cause

Sources/MLXInferenceCore/ModelArchitectureProbe.swift includes:

"qwen3_5",
"qwen3.5",
"qwen3_5_moe",

inside knownVisionModelTypes.

Because supportsVision checks membership in this set, the model_type alone is sufficient to force VLM mode.

This is ambiguous in the same way already documented in the file for Gemma 4: the same architecture/model type can exist in both text-only and multimodal forms, so model_type alone should not force VLM routing.

Proposed fix

Remove the three Qwen3.5/Qwen3.6 entries from knownVisionModelTypes and rely on the existing vision-specific indicators:

• vision_config
• known vision processor
• image_processor_type

This preserves VLM detection for genuine multimodal checkpoints while allowing text-only Qwen3.5/Qwen3.6 MoE models to use the LLM loader.

Patch attached: swiftlm-qwen35-moe-vlm-detection.patch

[traderjoe1968]

Proposed patch:

diff --git a/Sources/MLXInferenceCore/ModelArchitectureProbe.swift b/Sources/MLXInferenceCore/ModelArchitectureProbe.swift
index 19b2d47..3b00bd3 100644
--- a/Sources/MLXInferenceCore/ModelArchitectureProbe.swift
+++ b/Sources/MLXInferenceCore/ModelArchitectureProbe.swift
@@ -22,9 +22,6 @@ public enum ModelArchitectureProbe {
         "qwen2.5-vl",
         "qwen3_vl",
         "qwen3-vl",
-        "qwen3_5",
-        "qwen3.5",
-        "qwen3_5_moe",
         "idefics3",
         "gemma3",
         // "gemma4" deliberately absent: this codebase registers that exact model_type

[traderjoe1968]

Updated proposed patch after identifying the additional false-positive from preprocessor_config.json:

diff --git a/Sources/MLXInferenceCore/ModelArchitectureProbe.swift b/Sources/MLXInferenceCore/ModelArchitectureProbe.swift
index 19b2d47..3b86d43 100644
--- a/Sources/MLXInferenceCore/ModelArchitectureProbe.swift
+++ b/Sources/MLXInferenceCore/ModelArchitectureProbe.swift
@@ -22,9 +22,6 @@ public enum ModelArchitectureProbe {
         "qwen2.5-vl",
         "qwen3_vl",
         "qwen3-vl",
-        "qwen3_5",
-        "qwen3.5",
-        "qwen3_5_moe",
         "idefics3",
         "gemma3",
         // "gemma4" deliberately absent: this codebase registers that exact model_type
@@ -83,11 +80,29 @@ public enum ModelArchitectureProbe {
         let processorClass = preprocessor?["processor_class"] as? String
         let normalizedModelType = modelType?.lowercased().replacingOccurrences(of: ".", with: "_")
 
-        let supportsVision =
-            (normalizedModelType.map { knownVisionModelTypes.contains($0) } ?? false)
-            || (processorClass.map { knownVisionProcessors.contains($0) } ?? false)
-            || config["vision_config"] != nil
-            || preprocessor?["image_processor_type"] != nil
+        // Qwen3.5/Qwen3.6 model_type values are shared by text-only and
+        // multimodal checkpoints. Some text-only converted checkpoints also
+        // retain a stale Qwen3VL preprocessor_config.json. For these models,
+        // vision_config is authoritative; otherwise the stale processor
+        // metadata incorrectly routes the model through VLMModelFactory.
+        let ambiguousQwen35Types: Set<String> = [
+            "qwen3_5",
+            "qwen3.5",
+            "qwen3_5_moe",
+        ]
+
+        let supportsVision: Bool
+        if let normalizedModelType,
+           ambiguousQwen35Types.contains(normalizedModelType)
+        {
+            supportsVision = config["vision_config"] != nil
+        } else {
+            supportsVision =
+                (normalizedModelType.map { knownVisionModelTypes.contains($0) } ?? false)
+                || (processorClass.map { knownVisionProcessors.contains($0) } ?? false)
+                || config["vision_config"] != nil
+                || preprocessor?["image_processor_type"] != nil
+        }
 
         return ModelArchitectureInfo(
             modelType: modelType,

[solderzzc]

Thanks for the excellent root-cause writeup — exact same ambiguity class as #152's gemma4 fix, and your proposed fix is what's applied. PR up here: #157, watching CI now.

[solderzzc]

Merged: SwiftLM#157. All 14 CI jobs green, including speculative-decoding — confirmed this doesn't interact with the earlier gemma4/qwen3_5 auto-detect guard from #152. Thanks again for the thorough diagnosis and fix proposal.