Skip to content

[Bug Report] Qwen3.5/Qwen3Next expose query-gate rows as double-width W_Q #1652

Description

@emerardd

Describe the bug

Qwen3.5 and Qwen3Next expose a double-width W_Q through TransformerBridge. Their Hugging Face q_proj interleaves query and query-gate rows per head, so the public weight-analysis surface should expose only the query half with W_Q.shape[-1] == cfg.d_head.

The adapters try to do this in Qwen3ArchitectureAdapter._preprocess_gated_q_proj(), but the matcher only accepts keys ending in .self_attn.q_proj.weight. The real TransformerBridge.process_weights() path calls self.state_dict() first, which already exposes TransformerLens keys such as blocks.3.attn.q.weight. No key matches, preprocessing is a silent no-op, and AttentionBridge.W_Q interprets query-plus-gate rows as a query head twice the configured width.

This affects Qwen3.5, Qwen3Next, and the Qwen3.5 multimodal adapter that reuses the same helper. Ordinary Qwen3 is not gated and is unaffected.

This is silent in normal forward-parity tests because the underlying Hugging Face projection correctly uses both halves. However, composition scores, QK analysis, SVD, weight-space circuit analysis, and other code that trusts W_Q.shape[-1] == cfg.d_head consume the wrong geometry.

Code example

Using tiny random models constructed from config only (no checkpoint or tokenizer download) on current dev-4.x (ac4f7f134b12):

qwen3_5
  state key: blocks.3.attn.q.weight
  raw q:     (256, 128)
  W_Q:       (2, 4, 128, 64)
  d_head:    32

qwen3_next
  state key: blocks.3.attn.q.weight
  raw q:     (256, 128)
  W_Q:       (2, 4, 128, 64)
  d_head:    32

The existing preprocessing unit tests use synthetic Hugging Face-style keys such as model.layers.0.self_attn.q_proj.weight, so they pass without exercising the key format used by the real Bridge path.

Suggested scope:

  • Make the public W_Q surface return the per-head query rows only for gated-q-projection architectures, without mutating or discarding the live query-gate projection.
  • Add end-to-end tiny Qwen3.5 and Qwen3Next regression coverage through build_bridge_from_module().
  • Use distinguishable query/gate rows to verify that W_Q contains query rows rather than merely having the right shape.
  • Keep ordinary Qwen3 as a non-gated control.
  • Preserve Hugging Face forward parity and the hook_q_gate surface.

System Info

  • Installed from source with the repository uv environment
  • Windows 11 / PowerShell
  • Python 3.12.10
  • TransformerLens dev-4.x commit ac4f7f134b12

Additional context

Changing only the matcher to accept blocks.N.attn.q.weight is risky: the current processing pipeline distributes processed tensors back into generalized components, so slicing the live q_proj can remove the gate from model execution. The regression should cover both the analysis tensor and runtime behavior.

Checklist

  • I have checked that there is no similar issue in the repo (required)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions