Describe the bug
Qwen3.5 and Qwen3Next expose a double-width W_Q through TransformerBridge. Their Hugging Face q_proj interleaves query and query-gate rows per head, so the public weight-analysis surface should expose only the query half with W_Q.shape[-1] == cfg.d_head.
The adapters try to do this in Qwen3ArchitectureAdapter._preprocess_gated_q_proj(), but the matcher only accepts keys ending in .self_attn.q_proj.weight. The real TransformerBridge.process_weights() path calls self.state_dict() first, which already exposes TransformerLens keys such as blocks.3.attn.q.weight. No key matches, preprocessing is a silent no-op, and AttentionBridge.W_Q interprets query-plus-gate rows as a query head twice the configured width.
This affects Qwen3.5, Qwen3Next, and the Qwen3.5 multimodal adapter that reuses the same helper. Ordinary Qwen3 is not gated and is unaffected.
This is silent in normal forward-parity tests because the underlying Hugging Face projection correctly uses both halves. However, composition scores, QK analysis, SVD, weight-space circuit analysis, and other code that trusts W_Q.shape[-1] == cfg.d_head consume the wrong geometry.
Code example
Using tiny random models constructed from config only (no checkpoint or tokenizer download) on current dev-4.x (ac4f7f134b12):
qwen3_5
state key: blocks.3.attn.q.weight
raw q: (256, 128)
W_Q: (2, 4, 128, 64)
d_head: 32
qwen3_next
state key: blocks.3.attn.q.weight
raw q: (256, 128)
W_Q: (2, 4, 128, 64)
d_head: 32
The existing preprocessing unit tests use synthetic Hugging Face-style keys such as model.layers.0.self_attn.q_proj.weight, so they pass without exercising the key format used by the real Bridge path.
Suggested scope:
- Make the public
W_Q surface return the per-head query rows only for gated-q-projection architectures, without mutating or discarding the live query-gate projection.
- Add end-to-end tiny Qwen3.5 and Qwen3Next regression coverage through
build_bridge_from_module().
- Use distinguishable query/gate rows to verify that
W_Q contains query rows rather than merely having the right shape.
- Keep ordinary Qwen3 as a non-gated control.
- Preserve Hugging Face forward parity and the
hook_q_gate surface.
System Info
- Installed from source with the repository
uv environment
- Windows 11 / PowerShell
- Python 3.12.10
- TransformerLens
dev-4.x commit ac4f7f134b12
Additional context
Changing only the matcher to accept blocks.N.attn.q.weight is risky: the current processing pipeline distributes processed tensors back into generalized components, so slicing the live q_proj can remove the gate from model execution. The regression should cover both the analysis tensor and runtime behavior.
Checklist
Describe the bug
Qwen3.5 and Qwen3Next expose a double-width
W_QthroughTransformerBridge. Their Hugging Faceq_projinterleaves query and query-gate rows per head, so the public weight-analysis surface should expose only the query half withW_Q.shape[-1] == cfg.d_head.The adapters try to do this in
Qwen3ArchitectureAdapter._preprocess_gated_q_proj(), but the matcher only accepts keys ending in.self_attn.q_proj.weight. The realTransformerBridge.process_weights()path callsself.state_dict()first, which already exposes TransformerLens keys such asblocks.3.attn.q.weight. No key matches, preprocessing is a silent no-op, andAttentionBridge.W_Qinterprets query-plus-gate rows as a query head twice the configured width.This affects Qwen3.5, Qwen3Next, and the Qwen3.5 multimodal adapter that reuses the same helper. Ordinary Qwen3 is not gated and is unaffected.
This is silent in normal forward-parity tests because the underlying Hugging Face projection correctly uses both halves. However, composition scores, QK analysis, SVD, weight-space circuit analysis, and other code that trusts
W_Q.shape[-1] == cfg.d_headconsume the wrong geometry.Code example
Using tiny random models constructed from config only (no checkpoint or tokenizer download) on current
dev-4.x(ac4f7f134b12):The existing preprocessing unit tests use synthetic Hugging Face-style keys such as
model.layers.0.self_attn.q_proj.weight, so they pass without exercising the key format used by the real Bridge path.Suggested scope:
W_Qsurface return the per-head query rows only for gated-q-projection architectures, without mutating or discarding the live query-gate projection.build_bridge_from_module().W_Qcontains query rows rather than merely having the right shape.hook_q_gatesurface.System Info
uvenvironmentdev-4.xcommitac4f7f134b12Additional context
Changing only the matcher to accept
blocks.N.attn.q.weightis risky: the current processing pipeline distributes processed tensors back into generalized components, so slicing the liveq_projcan remove the gate from model execution. The regression should cover both the analysis tensor and runtime behavior.Checklist