Skip to content

Fix Qwen3.5 multimodal packing kwargs compatibility - #9838

Draft
taking-lying-flat wants to merge 1 commit into
modelscope:mainfrom
taking-lying-flat:agent/fix-qwen35-causal-conv-routing
Draft

Fix Qwen3.5 multimodal packing kwargs compatibility#9838
taking-lying-flat wants to merge 1 commit into
modelscope:mainfrom
taking-lying-flat:agent/fix-qwen35-causal-conv-routing

Conversation

@taking-lying-flat

@taking-lying-flat taking-lying-flat commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Summary

  • prevent Qwen3.5 text packing kwargs (cu_seq_lens_q, cu_seq_lens_k, max_length_q, and max_length_k) from leaking into the vision tower
  • register the filter as an instance-scoped, idempotent vision forward pre-hook for both dense and MoE loaders
  • preserve the same packing kwargs for the language model path
  • set the dense Qwen3.5 and OvisOCR2 minimum Transformers version to >=5.2.0, matching the first stable Qwen3.5 release and the existing MoE requirement

Root cause

Transformers 5.9+ forwards the model-level packing kwargs into the Qwen3.5 vision tower. The vision attention path also supplies its own sequence-length kwargs explicitly, which raises a duplicate-keyword TypeError. The hook strips only the four text packing keys at the vision boundary while leaving the original kwargs unchanged for the language model.

The previous dense/Ovis requirement (transformers>=5.0.0.dev) also admitted stable 5.0/5.1 releases that do not provide Qwen3.5. The corrected lower bound is 5.2.0.

Compatibility

Tested every published stable Transformers release in the project's supported range:

5.2.0, 5.3.0, 5.4.0, 5.5.0, 5.6.0, 5.7.0, 5.8.0, 5.9.0, 5.10.0, 5.11.0, 5.12.0, 5.12.1, 5.13.0, 5.13.1, 5.14.0, and 5.14.1.

The collision begins in 5.9.0. On 5.2-5.8 the hook remains compatible and harmless.

Local validation

  • external compatibility harness (not included in this PR): all checks passed on each of the 16 supported stable Transformers releases
  • Transformers 5.12.1 and 5.14.1 + Torch 2.13.0+cu130 + FlashAttention 2.8.3 + CUDA: image/video forward and backward passed
  • project pre-commit checks: flake8, isort, yapf, whitespace, EOF, quote, merge-conflict, and line-ending checks passed
  • git diff --check passed

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@taking-lying-flat
taking-lying-flat force-pushed the agent/fix-qwen35-causal-conv-routing branch from 90b6695 to 0622737 Compare August 3, 2026 18:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant