Add lookup-table instructions (LUTI) Learning Path

content/learning-paths/mobile-graphics-and-gaming/luti/01_overview.md

@@ -0,0 +1,101 @@
+---
+title: Understand why LUTI helps low-bit matrix multiplication
+description: Understand how packed low-bit weights reduce memory traffic and how LUTI expands their indices inside an Arm vector data path.
+weight: 2
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Overview
+We are at a point where Large Language Model(LLM) inference on the CPU is practical on mobile and edge devices. This is due in large part to the rise of low-bit AI models.
+Low-bit AI models store weights in compact packed format that must be efficiently expanded before matrix multiplication. You will see how 2-bit and 4-bit weights are stored, why conventional unpacking adds cycles, and how lookup-table instructions (LUTI) removes the unpacking step.
+
+By the end, you should be able to explain why packed low-bit indices are efficient for storage and memory traffic, how the indices are laid out in memory and how to use LUTI instructions to efficiently expand it.
+
+## Why use sub-byte weights?
+
+Large language model (LLM) inference on mobile and edge devices is often limited by memory capacity and bandwidth. During inference, model weights must be transferred from memory to the CPU, contributing to latency and energy use.
+
+Quantization reduces this traffic by storing weights in lower-precision formats. Weight-only quantization maps each 32-bit floating-point (`fp32`) weight to a compact logical code and stores shared metadata, such as a scale or zero point, for each block.
+
+{{% notice Note %}} The terms `4-bit` and `2-bit` specify the number of bits assigned to each logical code. These codes do not necessarily denote the numerical datatypes `int4` or `int2`. A `4-bit` or `2-bit` code might represent a signed integer, an unsigned integer, or an index into a codebook, depending on the quantization format. {{% /notice %}}
+
+Physical packing is the storage layout that places several low-bit codes into each byte. If you ignore the metadata, four `2-bit` codes or two `4-bit` codes can be stored in one byte.
+
+<p align="center">
+  <img
+    src="../images/luti_datatypes.png"
+    alt="Data Type Storage in Vector Register"
+    width="85%"
+  />
+</p>
+
+<p align="left">
+  <em>Figure 1. Packing low-bit weight codes into a scalable vector register. An int8 value occupies 1 byte, whereas one byte can hold four 2-bit codes or two 4-bit codes. Consequently, a 128-bit (16 bytes) vector can contain 16 int8 values, 32 packed 4-bit codes, or 64 packed 2-bit codes.
+</em>
+</p>
+
+This approach trades reconstruction accuracy for lower memory use. Its value also depends on decoding the packed codes efficiently. LUTI addresses that work by expanding low-bit codes directly into arithmetic-ready vector values.
+
+## Understand the LUTI operation
+
+Matrix multiplication kernels do not usually operate on packed 2-bit or 4-bit codes.
+Before arithmetic, the codes must be decoded into values the computation can consume.
+
+Conceptually, the operation is:
+```c
+index = get_lut_index(packed_code);
+expanded_value = lookup_table[index];
+```
+
+Armv9-A LUTI instructions perform lookup-table operations that map low-bit indices to expanded values. LUTI2 and LUTI4 operate on 2-bit and 4-bit indices, respectively.
+
+- `LUTI2` uses each 2-bit index to select one of four lookup-table values.
+- `LUTI4` uses each 4-bit index to select one of sixteen lookup-table values.
+
+The lookup table defines the expanded value associated with each code according to the quantization scheme.
+
+### LUT for 2-bit codes
+
+For example, a 2-bit lookup table might contain:
+
+| Packed Code | LUT Index | Expanded Value |
+|---|---|---|
+| `0b00` | lut[0] | `-2` |
+| `0b01` | lut[1] | `-1` |
+| `0b10` | lut[2] | `0`  |
+| `0b11` | lut[3] | `1`  |
+
+
+### From packed 2-bit codes to 8-bit values
+The key benefit of LUTI is that the matrix multiplication kernel can load weights in their compact form. A source vector of packed weights therefore carries more values per memory load than a vector containing already expanded 8-bit, 16-bit, or 32-bit values.
+
+For 2-bit codes, LUTI uses the packed indices in a source vector to select lookup-table entries and writes the resulting values to destination vector registers. Figure 2 shows how 2-bit codes expand into 8-bit values.
+
+<p align="center">
+  <img
+    src="../images/luti_flow_overview.png"
+    alt="Lookup-Table Overview"
+    width="90%"
+  />
+</p>
+
+<p align="left">
+  <em>Figure 2. LUTI maps packed 2-bit codes to lookup-table indices, selects the relevant 8-bit values, and writes them to a destination vector. Compared with storing expanded 8-bit values, the packed format allows each memory load to supply four times as many weights.
+</em>
+</p>
+
+## Identify LUTI responsibilities
+
+Keep these boundaries in mind when using LUTI:
+
+- The lookup table defines the meaning of each packed code
+- LUTI2 and LUTI4 expand indices; they don't calculate quantization metadata
+- Scaling, zero-point correction, bias, activation, clamping, and requantization remain separate operations
+- Expansion happens in the vector path, close to the arithmetic that consumes the values
+
+## What you've learned and what's next
+You've learned how LUTI uses packed low-bit codes as indices and expands them into values for subsequent arithmetic.
+
+Next, you'll set up the compiler and SME2 hardware needed to build and run the examples.

content/learning-paths/mobile-graphics-and-gaming/luti/02_luti_and_sme2.md

@@ -0,0 +1,148 @@
+---
+title: Use LUTI with SME2
+description: Trace packed indices through the SME2 ZT0 table, streaming Z registers, and ZA matrix accumulators.
+weight: 3
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## SME and SME2 architectural features
+
+SME extends the Armv9-A architecture and accelerates matrix-heavy computations, such as outer products and matrix multiplication.
+SME introduces __Streaming SVE mode__ and the __scalable ZA matrix-storage__ array. ZA accumulates matrix outer products and multi-vector dot products.
+
+SME2 builds on SME and adds __multi-vector instructions__ and the fixed __512-bit ZT0 lookup-table register__.
+
+For this learning path, the important points to note is that LUTI uses packed low-bit codes from Z source registers, reads the corresponding look-up table entries from `ZT0`, and writes expanded operands into Z destination registers.
+The expanded operands can then be consumed by SME2 matrix instructions such as `SDOT` or `SMOPA`, with results accumulated in `ZA` array.
+
+## ZT0 lookup-table register
+
+SME2 provides a fixed 512-bit architectural register named `ZT0`. It contains 64 bytes, arranged as sixteen 32-bit table entries.
+
+<p align="center">
+  <img
+    src="../images/luti2.png"
+    alt="SME2 ZT0 Lookup-Table Register"
+    width="100%"
+  />
+    <img
+    src="../images/luti4.png"
+    alt="SME2 ZT0 Lookup-Table Register"
+    width="100%"
+  />
+</p>
+
+<p align="left">
+  <em>Figure 3. ZT0 lookup-table organization and use by LUTI2 and LUTI4. ZT0 entries form a single linear table of entries 0–15. LUTI2 selects among entries 0–3 using 2-bit indices, while LUTI4 can select among all entries 0–15 using 4-bit indices. According to the destination element size, LUTI copies the low 8, 16, or 32 bits of the selected 32-bit table entry into the destination Z registers.
+</em>
+</p>
+
+LUTI instructions use packed low-bit indices from a source Z register (`Zn`) to select the corresponding `ZT0` register entry.
+
+The relevant `ZT0` entries are expanded to chosen output element destination width and written to output Z registers (`Zd`).
+
+LUTI2 and LUTI4 can populate one, two, or four destination Z registers. The number of destination registers and the expanded element width (`.B`, `.H`, or `.S`) determine how many packed source bits fill the destinations.
+
+The element suffix specifies the expanded destination width:
+- `.B` produces 8-bit elements.
+- `.H` produces 16-bit elements.
+- `.S` produces 32-bit elements.
+
+## Streaming mode
+
+ZT0-based SME2 LUTI instructions need both streaming mode and ZA enabled. `SMSTART` enables the required state, and `SMSTOP` disables it.
+
+Streaming mode changes the execution context in three ways:
+
+- Vector and predicate lengths use the *streaming vector length* (SVL), which can differ from the non-streaming vector length
+- Streaming instructions, including the multi-register LUTI2, SDOT, and SMOPA forms, become available
+- `PSTATE.ZA` controls access to both the ZA matrix-storage array and `ZT0`
+
+Efficient kernels enter streaming mode before repeated loops and exit afterwards. Streaming mode does not automatically stream matrix data from memory. The kernel still loads only the current computation tile.
+
+## Follow the SME2 LUTI data path
+
+A simplified SME2 LUTI sequence is:
+
+```text
+SMSTART
+* load ZT0 once
+* perform LUTI2
+SMSTOP
+```
+
+A detailed LUTI SME2 flow is:
+  1. Enter SME streaming mode with `SMSTART` and load the LUT into the sixteen 32-bit `ZT0` register.
+  2. Load LHS activations and packed RHS data for the current computation tile.
+  3. Use LUTI2 or LUTI4 to expand the packed RHS indices from `ZT0` into Z registers.
+  4. Feed the expanded RHS elements and LHS activations to SME2 instructions, such as `SDOT` or `SMOPA`.
+  5. Accumulate partial matrix products in the SME2 `ZA` array.
+  6. Convert, clamp, and store the completed output tile as required by the kernel.
+  7. Exit SME streaming mode with `SMSTOP`. This disables the ZA and ZT0 state after the kernel completes.
+
+LUTI replaces the explicit unpack/decode portion of the data path. It does not replace the matrix multiply instruction that consumes the expanded values.
+
+### Example: kernel with LUTI2
+
+This example shows how LUTI2 expands packed 2-bit RHS weights. It simplifies register allocation, predication, addressing, and loop control to focus on the LUTI data flow.
+
+{{% notice Note %}} This example uses a 512-bit streaming vector length (SVL). The SVL is a CPU specific property. {{% /notice %}}
+
+__1. Define and pass the LUT__
+
+`ZT0` contains sixteen 32-bit entries. LUTI2 uses entries 0–3; entries 4–15 are unused and contain zero.
+```c
+static const int32_t lut_i8_i2[16] = {-2, -1, 0, 1,};
+```
+
+For a `.B` LUTI result, the low 8 bits of the selected 32-bit entry form the destination element.
+
+__2. Load the LUT into ZT0__
+
+Enter streaming mode, initialize ZA, and load ZT0. The lookup table does not change across the inner matrix loop, so load it once before the loop.
+
+```asm
+smstart                   // Enable Streaming SVE mode and ZA/ZT0 state
+zero    {za}              // Zero initialize accumulators for this output tile
+ldr     zt0, [x_lut]      // load LUT into fixed 512-bit ZT0 table
+```
+
+__3. Load LHS and packed RHS__
+
+Load the LHS activations and packed RHS 2-bit indices for the current computation tile, not the entire matrix.
+
+```asm
+ld1rqb  {z0.b}, ... , [x_lhs]               // load LHS
+ld1b    {z16.b-z19.b}, ... , [x_packed_rhs] // load RHS packed 2-bit indices
+```
+  - The `z0.b` register receives the LHS activations.
+  - The `z16.b`–`z19.b` registers receive the packed RHS 2-bit indices.
+
+`ld1rqb` is the SVE load-and-replicate-quadword operation. For a 512-bit SVL, you can view `z0` as four 128-bit regions. `ld1rqb` replicates the 16-byte LHS block across those regions.
+
+__4. LUTI2 expands the packed indices__
+```asm
+luti2   { z24.b - z27.b }, zt0, z16[0]     // unpack 2-bit indices
+luti2   { z4.b  - z7.b  }, zt0, z17[0]
+
+luti2   { z8.b  - z11.b }, zt0, z18[0]
+luti2   { z12.b - z15.b }, zt0, z19[0]
+```
+Each LUTI2 instruction reads packed 2-bit indices from one source Z register (`z16` to `z19`) and expands them into four `.B` destination registers for a 512-bit SVL.
+
+__5. Feed the expanded vectors directly to SDOT__
+
+The expanded vectors can now feed SME2 matrix instructions such as `SDOT` or `SMOPA`, which accumulate the results in `ZA`.
+
+## What you've learned and what's next
+
+You've learned how LUTI operates within SME2: a micro-kernel loads the lookup table into `ZT0`, uses packed low-bit indices in Z registers to expand the RHS values, and feeds those expanded values into SME2 instructions.
+
+Next, you'll apply these concepts to a complete low-bit matrix multiplication kernel and examine how LUTI can replace explicit unpacking and decoding in the inner computation loop.
+
+## Further reading
+- [Arm SME2 Introduction](https://developer.arm.com/community/arm-community-blogs/b/architectures-and-processors-blog/posts/part4-arm-sme2-introduction)
+- [SME2 lookup table Armv9-A Documentation](https://developer.arm.com/documentation/109246/0101/SME-Overview/SME-and-SME2/SME2-lookup-table)
+- [Introduction to streaming and non-streaming mode](https://arm-software.github.io/acle/main/acle.html#controlling-the-use-of-streaming-mode)