[Bug]: I2_S GEMM fast path produces garbage for multi-token prompts on AVX-only CPUs

[puddintaim1975-dev]

Description

On CPUs without AVX2 (e.g., Intel Xeon E5-2690 v2, Ivy Bridge, AVX-only), the I2_S GEMM fast path in ggml-cpu.c:1492 produces corrupt output when processing prompts with more than ~3 tokens. Single-token generation (GEMV path) works correctly.

Symptoms

• 1-3 token prompts → coherent output
• 5+ token prompts → ?????? or garbled output
• The corruption affects the KV cache: even after the prompt is processed, subsequent generation tokens are garbled

Root Cause

The GEMM fast path (ggml_gemm_i2_i8_s) is called for multi-token prompt evaluation (when src1 has multiple columns). The scalar fallback implementation has a bug in how it indexes the I2_S weight matrix and/or activation matrix for column strides > 1.

Workaround

Disabling the GEMM fast path forces I2_S through the dequantize-then-float-matmul path:

// ggml-cpu.c:1492 — change from:
if (src0->type == GGML_TYPE_I2_S && ggml_n_dims(src0) == 2) {
// to:
if (false && src0->type == GGML_TYPE_I2_S && ggml_n_dims(src0) == 2) {

This produces correct results but is slower (~0.6 tok/s prompt eval vs ~26 tok/s for F16 on the same hardware).

Key Distinction from #547

This is distinct from #547/PR #580 which covers the empty-body scalar fallback for ggml_vec_dot_i2_i8_s_* kernels. After applying those fixes (or equivalent scalar implementations), the GEMM path still produces wrong results for multi-token prompts.

The GEMV path (line ~1217) works correctly for single-token generation. The issue is specifically in ggml_gemm_i2_i8_s when called with nr > 1 (multiple activation columns).

Reproduction

# Build with AVX2 disabled (forces scalar fallback)
cmake -B build -DBITNET_ARM_TL1=OFF -DBITNET_X86_TL2=OFF
cmake --build build --target llama-server -j8

# Short prompt works
curl http://localhost:8081/v1/completions \
  -d '{"model":"bitnet","prompt":"Hello","max_tokens":20}'
# → " there! I'm happy to help" ✓

# Longer prompt fails
curl http://localhost:8081/v1/completions \
  -d '{"model":"bitnet","prompt":"The weather today is","max_tokens":20}'
# → "?????" ✗

Also requires fixes #588 (ReLU²) and PR #616 (weight scale direction) for coherent F16 output.

Environment

• CPU: Intel Xeon E5-2690 v2 (Ivy Bridge, AVX only, no AVX2)
• OS: Ubuntu 24.04 LTS
• Compiler: Clang 18.1.3
• CMake flags: -DBITNET_ARM_TL1=OFF -DBITNET_X86_TL2=OFF
• Model: BitNet-b1.58-2B-4T (I2_S format)
• BitNet commit: 390c30775

[shifulegend]

That is a nasty bug. If you are stuck debugging the multi-token GEMM path on older AVX hardware and just need a working baseline for the model right now, one alternative worth looking at is Project Zero. It is a pure C99 engine that runs BitNet natively. It basically skips the whole ggml integration layer entirely. On a Xeon it hits 36 tok/s, and the fallback path actually runs fine on older silicon without producing that garbled output on longer prompts. There is a pre-built Linux binary in the releases if you just want to verify the model output.