Fix: ternary weight dequantization uses division instead of multiplication

[puddintaim1975-dev]

Bug

The converter dequantizes ternary weights for F16/F32 GGUF output by dividing by weight_scale instead of multiplying. This produces weight values approximately 2.4× too small, resulting in completely garbled output from F16/F32 GGUF conversions.

Root Cause

BitNet uses absmean quantization: weight_scale = mean(|W|) per output row. The dequantization should reconstruct original weight magnitudes by multiplying ternary values by this scale:

Correct:  ternary_value * weight_scale  →  {-1.555, 0, 1.555}  (scale ≈ 1.555)
Wrong:    ternary_value / weight_scale  →  {-0.643, 0, 0.643}

Why both paths need * (not just BitnetModel)

The LlamaModel path (line 807) and BitnetModel path (line 1108) both process the same weight_scale format from the checkpoint. weight_scale is defined by AutoBitLinear in the HuggingFace transformers implementation as mean(|W|) per row — the direct mean, not its inverse. The offline dequantization formula is ternary * weight_scale, verified in src/transformers/integrations/bitnet.py:

# AutoBitLinear.forward() (offline mode, line 292):
output = F.linear(input, weight) * self.weight_scale

# where weight = unpacked ternary {-1, 0, +1}
# and self.weight_scale = mean(|W|) per row

No known model uses weight_scale = 1/mean(|W|) (the inverse). The 1bitLLM models (1bitLLM/bitnet_b1_58-large, 1bitLLM/bitnet_b1_58-3B) use online quantization with BitLinear and have no weight_scale tensors at all. The only model with weight_scale is microsoft/bitnet-b1.58-2B-4T, which uses BitnetForCausalLM → BitnetModel path.

Changes

Two lines in utils/convert-hf-to-gguf-bitnet.py:

• Line 807 (LlamaModel.write_tensors): data_torch / scale_map → data_torch * scale_map
• Line 1108 (BitnetModel.write_tensors): data_torch / scale_map → data_torch * scale_map
• Line 1105: Updated comment from "divide by" to "multiply by"

Numerical Proof

For model.layers.0.mlp.gate_proj:

• weight_scale = 1.5547 (= mean(|W|) of original BF16 weight)
• Ternary values: {-1, 0, +1}, 60.87% nonzero

| Operation | Range | mean(|W|) | Verdict |
|-----------|-------|-------------|---------|
| ternary * scale | [-1.5547, 1.5547] | 0.9463 | ✓ Correct |
| ternary / scale | [-0.6432, 0.6432] | 0.3915 | ✗ 2.42× too small |

Verification

Tested on BitNet-b1.58-2B-4T (commit 390c30775):

Metric	Division (broken)	Multiplication (fixed)
F16 weight magnitude	~0.64	~1.55
Coherent text output	✗	✓
"The capital of France is"	Garbled	"Paris. Paris is a city known for its rich history..."

Note: I2_S output format is unaffected — ternary values are stored directly and scale is passed separately to quantize_to_i2_s.

Environment

• CPU: Intel Xeon E5-2690 v2 (Ivy Bridge, AVX only)
• Build: Clang 18.1.3, -DBITNET_ARM_TL1=OFF -DBITNET_X86_TL2=OFF

Also required for correct F16 output: Fix #588 (LLM_FFN_SILU → LLM_FFN_RELU_SQR).

[puddintaim1975-dev]

Reference Implementation Proof

The HuggingFace transformers AutoBitLinear (offline quantization mode) confirms multiplication is correct:

src/transformers/integrations/bitnet.py:292:

# AutoBitLinear.forward() for offline-quantized models:
output = F.linear(input, weight) * self.weight_scale
# weight = unpacked ternary {-1, 0, +1}
# self.weight_scale = mean(|W|) per row (stored in checkpoint)

The formula is: dequantized = ternary * mean(|W|), not ternary / mean(|W|).

The online quantization path (BitLinear/WeightQuant) uses s = 1/mean(|W|) internally, but this inverse scale is never stored in the checkpoint. The checkpoint weight_scale for offline-quantized models is always the direct mean, matching the AutoBitLinear convention.

Model format check:

• microsoft/bitnet-b1.58-2B-4T: BitnetForCausalLM, linear_class: autobitlinear, quantization_mode: offline → has weight_scale tensors, uses BitnetModel path
• 1bitLLM/bitnet_b1_58-large: BitnetForCausalLM, no quantization_config → online quantization, no weight_scale tensors
• 1bitLLM/bitnet_b1_58-3B: Same as above, no weight_scale tensors

No known model stores weight_scale as 1/mean(|W|). Both converter paths (LlamaModel line 807 and BitnetModel line 1108) encounter the same format and both need * not /.