[michalharakal]

Summary

A Q8_0 or Q4_0 Linear can now be exported with its weight kept packed: integer codes plus per-block scales, emitted as a linalg chain that an NPU compiler can match to a block-quantized matmul kernel. Until now a packed weight could only fail the export or be dequantized to a dense FP32 constant.

• PackedConstantHandling.KEEP_PACKED (skainet-compile-dag): the weight node carries its PackedBlockStorage under packed_weight, with no float copy. HloGenerator.generate takes packedConstants.
• BlockQuantMatmulFusionPass (skainet-compile-opt, DAG phase): folds weight → transpose → matmul (or a transposeB matmul, after TransposeEliminationPass) into one block_quant_matmul node. It only folds a weight that is exclusive, Q8_0/Q4_0, ROW_MAJOR and has in % 32 == 0; every skip is reported in the diagnostics.
• BlockQuantMatmulConverter (skainet-compile-hlo, registered in every converter set): emits (sitofp(codes) - zp) * scale over [nB, 32, N] i8/i4 codes with bf16 [nB, 1, N] scales, then collapse/expand → linalg.batch_matmul. The activation is converted to bf16 and reshaped to [1, M, K] around the chain, and all constants are inline.
  • A block with a negative scale d is stored exactly as codes -1 - c, zero point -1 and scale -d. That fits i8 and i4, so ggml Q4_0 (d = max / -8) needs no requantization. The zero-point tensor is emitted only when such a block exists.
  • All-zero blocks get scale 1.
• Unfolded weights fail loudly: a packed weight still reaching ConstantOperationsConverter fails the export and names the fusion pass and DEQUANTIZE.

A target opts in with TargetOptimizers.registerDagPasses("torq") { listOf(BlockQuantMatmulFusionPass()) }. The docs (compile-flexibility.adoc) and the CHANGELOG are updated.

Verification

• BlockQuantMatmulExportTest (common, 7 tests): it traces matmulWeightTransposed with real Q8_0/Q4_0 storage, then runs KEEP_PACKED, the pass and toStableHlo. It checks:
  • the chain is emitted with no dot_general;
  • the zero point appears for Q4_0;
  • batched rows are kept;
  • the regrouped weight decodes to storage.toFloatArray() within bf16 scale rounding;
  • an all-zero block gets a scale the kernel accepts;
  • an unfolded packed weight fails the export.
• Test suites pass on JVM: dag 117, opt 74, hlo 285. apiDump is updated, and hlo compiles for linuxX64 and wasmJs.
• TorqBlockQuantGoldenTest (jvm) runs only when SKAINET_TORQ_COMPILE names a command that takes torq-compile's arguments. With Torq 2.3.0 (--iree-input-type=stablehlo-torq --torq-hw=SL2610), both the Q8_0 and the Q4_0 (negative scales) projection compile to torq_hl.mixed_quant_matmul, 8 tiles each, with no generic torq_hl.matmul.
• On an SL2610 board, the vmfbs built from this test's emitted modules match bf16(x) · Wᵀ with the dequantized weight: cosine 0.999996 (Q8_0) and 0.999997 (Q4_0).

Limits

• The Torq kernel matches a one-row activation only. With M > 1 (prefill), the chain is still correct but runs on the generic path.
• A weight with a second consumer (a tied embedding / vocabulary projection) is not folded, so the export fails until that case gets its own handling.
• Stock IREE llvm-cpu has produced wrong results for i4 weights, so keep Q4_0 to targets that were checked.

🤖 Generated with Claude Code

[github-actions]

📖 Documentation Preview

The documentation has been built successfully for this PR.

Generated Files:

• Operator documentation: docs/modules/operators/_generated_/
• JSON schema output: operators.json

Artifacts:

• Download the documentation-preview-1478 artifact to view the complete documentation locally.

This comment will be updated automatically when the PR is updated.