From b578177efbf4be7e5899191af3091be23f74987d Mon Sep 17 00:00:00 2001 From: Michal Harakal <471011+michalharakal@users.noreply.github.com> Date: Fri, 6 Nov 2026 17:34:01 +0100 Subject: [PATCH] release: prepare 0.64.0 -- NPU decode-step export and LFM2, against engine 0.63.0 Version 0.64.0 in gradle.properties, the README release paragraph and the BOM snippets in README and docs; the LFM2 README names engine 0.63.0. The CHANGELOG Unreleased section becomes 0.64.0 with the LFM2 modules (#527), the engine pin, the llama test tolerance and the dependency updates. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 28 +++++++++++++++++-- README.md | 4 +-- .../how-to/export-moonshine-v2-streaming.adoc | 2 +- .../pages/reference/moonshine-encoder.adoc | 2 +- .../ROOT/pages/reference/typed-decisions.adoc | 2 +- .../tutorials/android-getting-started.adoc | 2 +- .../pages/tutorials/getting-started-java.adoc | 4 +-- .../pages/tutorials/getting-started-leaf.adoc | 2 +- .../pages/tutorials/llama3-tool-calling.adoc | 2 +- gradle.properties | 2 +- llm-inference/lfm2/README.md | 2 +- 11 files changed, 38 insertions(+), 14 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 05a584c6..a23e68aa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,12 +9,32 @@ requiring an engine release. `0.57.1` against engine `0.57.0` is such a release. The format roughly follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [Unreleased] +## [0.64.0] — 2026-11-06 -Against engine **0.63.0** (SKaiNET#1478 and SKaiNET#1479, block-quant export). +Against engine **0.63.0** (block-quantized weights through StableHLO export: SKaiNET#1478 and SKaiNET#1479, plus +the Unicode Split fix in the JVM tokenizer, SKaiNET#1477). Headline: **one decode step of a KV-cache decoder as an NPU +graph with block-quantized weights** — FunctionGemma 270M runs its decode step on the Torq NPU of an SL2610 in one +dispatch, 162 ms per step against about 610 ms per token on the CPU path, with the same top-1. Plus the **LFM2** +model family and its chat frontend. + +Version note: the with-past types moved to `llm-core` and the Gemma / `DecoderKvModel` nested classes are gone (see +Changed); code that constructs `ChunkContext` directly needs one change. Everything else is additive. ### Added +- **LFM2** (`llm-inference:lfm2`, `skainet-transformers-inference-lfm2`, JVM): the dense, bias-free LFM2 hybrid + (gated causal short convolution with a fixed `[D,K]` history, grouped-query attention with per-head Q/K RMS norms + before rotation, SiLU-gated MLP, tied head). `Lfm2Config.parse`, `loadLfm2Model` (single SafeTensors checkpoint, + BF16 / F16 / F32 sources, full weight-manifest verification, optional checkpoint SHA-256), `Lfm2Session` with + explicit per-layer hybrid state, and a fixed-shape StableHLO export (`Lfm2ExportHarness`, `Lfm2GenerationExport`). + `BALANCED_FP32` RMS reduction is the default. Validated against the reference implementation of LFM2.5-350M + (4,924,416 teacher-forced and greedy logit comparisons, exact bounded generation traces) on x86-64 and ARM64. + `Lfm2Family` is registered in `TransformersFamilies` (#527). +- **`llm-runtime:klfm2`** (`skainet-transformers-runtime-klfm2`, JVM): the LFM2 chat frontend. + `loadLfm2TextFrontend` byte-pins the validated LFM2.5-350M metadata by default (`Lfm2MetadataPins`; `pins = null` + accepts other checkpoints through structural validation), renders the checkpoint's chat template (only verified + template revisions, never relaxed), and generates greedily with streaming, stop tokens and per-request state + isolation. An IREE path (`Lfm2IreeChatCli`, `Lfm2IreeGenerationRuntime`) runs the exported graphs (#527). - **Decode step for an NPU with block-quantized weights** (`llm-core`, jvm): `DecodeStepExport.export(spec, dtype) { inputs, ctx -> … }` traces one single-token step of any KV-cache decoder with a static past, the embedding lookup left to the host, keeps Q8_0 / Q4_0 weights packed and folds them with the engine's `BlockQuantMatmulFusionPass`, and returns the StableHLO @@ -36,6 +56,10 @@ Against engine **0.63.0** (SKaiNET#1478 and SKaiNET#1479, block-quant export). `MultiHeadAttention.slidingWindow`. `RopeCosSin(cos, sin)` and `RopeCosSin(cosGlobal, sinGlobal, cosSliding, sinSliding)` keep the old call shapes; `ChunkContext` now takes a `RopeCosSin`. `:llm-inference:gemma` exposes `:llm-core` as `api`. +- Engine pin **SKaiNET 0.63.0** (`skainet-bom:0.63.0` re-exported by the BOM). The llama `ForwardScopeSteadyStateTest` + pins the scoped decode to four ulp of each step's largest logit instead of four ulp of each value: 0.63.0 changes + the accumulation order, so a logit of 0.0066 moves by 7.5e-9, 16 of its own ulps but under one ulp of the largest. +- Dependencies: Ktor 3.6.1 (#525), kotlinx-io 0.9.2 (#526), logback 1.6.6 (#523). ## [0.63.3] — 2026-10-28 diff --git a/README.md b/README.md index f2041c66..2e08e14e 100644 --- a/README.md +++ b/README.md @@ -115,7 +115,7 @@ Honest status — see the project-status note at the top of this README. ## Current release -The current release is **0.63.3**, against **SKaiNET 0.62.0**: the streaming Moonshine runtime can hand the decoder to an external native library too (`ext:|` as the prefill path), so with 0.63.2's external encoder the whole model can run on an accelerator outside IREE while the runtime keeps the streaming loop, greedy decoding and its guards. 0.63.2 handed the encoder side to an external library (`ext:` as the encoder path) while the decoder stays on the IREE device. 0.63.1 gave the streaming Moonshine runtime an incremental frontend (an optional `frontend_step.vmfb` takes most of the frontend out of the end-of-utterance flush, transcripts unchanged) and `IreeModuleRunner` runs one compiled IREE module on Android. 0.63.0 brought expert heads and routing for Laya — a decision head fine-tuned on top of the frozen encoder (`expert.safetensors` + `expert.json`) loads on its own with `LayaExpertLoader`, any number of heads share one encoder pass per question (`LayaDecisionModel.logitsFrom`), `LayaRouterSpec` + `LayaMixtureAgent` route a request by asking the base head what kind of text it is, and `LayaMixtureDecisionAdapter` puts the mixture behind the typed decision contract; verified against the published layaMOE evaluation. The ModernBERT-large `laya-typed-decisions` checkpoint loads (`position_embedding_type: absolute`). The whisper StableHLO export ships as `WhisperExportCli`. Engine 0.62.0 brings the JVM ↔ Kotlin/Wasm parity: every kernel on the wasm tier, resident activations, fused attention / norms / softmax, browser IO parity. The Laya Q8_0 lane on the desktop JVM with the native provider needs engine 0.62.1 (SKaiNET #1436). +The current release is **0.64.0**, against **SKaiNET 0.63.0**: `DecodeStepExport` (`llm-core`) traces one single-token step of any KV-cache decoder with a static past and keeps Q8_0 / Q4_0 weights packed for the engine's block-quant matmul fusion, so FunctionGemma 270M's decode step compiles to one dispatch on the Torq NPU of an SL2610 (162 ms per step, same top-1 as the f32 reference); the with-past types (`RopeCosSin`, `ChunkContext`, `KvStepOutput`) are shared by Gemma and the generic KV decoder in `sk.ainet.lang.nn.dsl.decoder`. LFM2 arrives as `llm-inference:lfm2` (dense hybrid short-conv + GQA model, verified checkpoint loading, fixed-shape StableHLO export) and the `llm-runtime:klfm2` chat frontend. 0.63.3 let the streaming Moonshine runtime hand the decoder to an external native library too (`ext:|` as the prefill path), so with 0.63.2's external encoder the whole model can run on an accelerator outside IREE while the runtime keeps the streaming loop, greedy decoding and its guards. 0.63.2 handed the encoder side to an external library (`ext:` as the encoder path) while the decoder stays on the IREE device. 0.63.1 gave the streaming Moonshine runtime an incremental frontend (an optional `frontend_step.vmfb` takes most of the frontend out of the end-of-utterance flush, transcripts unchanged) and `IreeModuleRunner` runs one compiled IREE module on Android. 0.63.0 brought expert heads and routing for Laya — a decision head fine-tuned on top of the frozen encoder (`expert.safetensors` + `expert.json`) loads on its own with `LayaExpertLoader`, any number of heads share one encoder pass per question (`LayaDecisionModel.logitsFrom`), `LayaRouterSpec` + `LayaMixtureAgent` route a request by asking the base head what kind of text it is, and `LayaMixtureDecisionAdapter` puts the mixture behind the typed decision contract; verified against the published layaMOE evaluation. The ModernBERT-large `laya-typed-decisions` checkpoint loads (`position_embedding_type: absolute`). The whisper StableHLO export ships as `WhisperExportCli`. Engine 0.62.0 brings the JVM ↔ Kotlin/Wasm parity: every kernel on the wasm tier, resident activations, fused attention / norms / softmax, browser IO parity. The Laya Q8_0 lane on the desktop JVM with the native provider needs engine 0.62.1 (SKaiNET #1436). It builds on **0.62.2**, against **SKaiNET 0.61.0**: a transformers-only fix, the Android runtime's streaming Moonshine libraries are rebuilt with the utterance-start fix (#500; 0.62.0 and 0.62.1 packed the old build, which garbles transcripts on device). It builds on **0.62.1**, a transformers-only fix on 0.62.0, whose JVM `LayaModelLoader.load(directory, …)` had become an extension function that existing callers did not import; it is a member again (`@JvmOverloads`, so Java callers keep the 0.61 signature). @@ -305,7 +305,7 @@ The recommended way to consume is via the BOM. It pins every published `skainet- ```kotlin dependencies { - implementation(platform("sk.ainet.transformers:skainet-transformers-bom:0.63.3")) + implementation(platform("sk.ainet.transformers:skainet-transformers-bom:0.64.0")) // Versions resolved from the BOM: implementation("sk.ainet.transformers:skainet-transformers-core") diff --git a/docs/modules/ROOT/pages/how-to/export-moonshine-v2-streaming.adoc b/docs/modules/ROOT/pages/how-to/export-moonshine-v2-streaming.adoc index a8867080..cc045e9a 100644 --- a/docs/modules/ROOT/pages/how-to/export-moonshine-v2-streaming.adoc +++ b/docs/modules/ROOT/pages/how-to/export-moonshine-v2-streaming.adoc @@ -32,7 +32,7 @@ From your own build, resolve the published module and run its main class: ---- val exportTool by configurations.creating dependencies { - exportTool(platform("sk.ainet.transformers:skainet-transformers-bom:0.63.3")) + exportTool(platform("sk.ainet.transformers:skainet-transformers-bom:0.64.0")) exportTool("sk.ainet.transformers:skainet-transformers-inference-moonshine") } tasks.register("exportMoonshine") { diff --git a/docs/modules/ROOT/pages/reference/moonshine-encoder.adoc b/docs/modules/ROOT/pages/reference/moonshine-encoder.adoc index c65d7f76..864a263f 100644 --- a/docs/modules/ROOT/pages/reference/moonshine-encoder.adoc +++ b/docs/modules/ROOT/pages/reference/moonshine-encoder.adoc @@ -24,7 +24,7 @@ a DSL decoder is future work. [source,kotlin] ---- dependencies { - implementation(platform("sk.ainet.transformers:skainet-transformers-bom:0.63.3")) + implementation(platform("sk.ainet.transformers:skainet-transformers-bom:0.64.0")) implementation("sk.ainet.transformers:skainet-transformers-inference-moonshine") } ---- diff --git a/docs/modules/ROOT/pages/reference/typed-decisions.adoc b/docs/modules/ROOT/pages/reference/typed-decisions.adoc index 3dc8a4f8..0e35bb9a 100644 --- a/docs/modules/ROOT/pages/reference/typed-decisions.adoc +++ b/docs/modules/ROOT/pages/reference/typed-decisions.adoc @@ -12,7 +12,7 @@ below put that on the JVM path with the same contract for every model. [source,kotlin] ---- dependencies { - implementation(platform("sk.ainet.transformers:skainet-transformers-bom:0.63.3")) + implementation(platform("sk.ainet.transformers:skainet-transformers-bom:0.64.0")) implementation("sk.ainet.transformers:skainet-transformers-decisions-api") // the contract, no engine dependency implementation("sk.ainet.transformers:skainet-transformers-decisions-runtime") // validation + acceptance policy implementation("sk.ainet.transformers:skainet-transformers-inference-laya") // Laya as a DecisionAdapter diff --git a/docs/modules/ROOT/pages/tutorials/android-getting-started.adoc b/docs/modules/ROOT/pages/tutorials/android-getting-started.adoc index bf28859d..48cb4918 100644 --- a/docs/modules/ROOT/pages/tutorials/android-getting-started.adoc +++ b/docs/modules/ROOT/pages/tutorials/android-getting-started.adoc @@ -55,7 +55,7 @@ dependencies { // self-registers it on ART at process start — nothing to call. runtimeOnly("sk.ainet.core:skainet-backend-jni-cpu") - implementation(platform("sk.ainet.transformers:skainet-transformers-bom:0.63.3")) + implementation(platform("sk.ainet.transformers:skainet-transformers-bom:0.64.0")) implementation("sk.ainet.transformers:skainet-transformers-core") implementation("sk.ainet.transformers:skainet-transformers-runtime-kllama") implementation("sk.ainet.transformers:skainet-transformers-inference-llama") diff --git a/docs/modules/ROOT/pages/tutorials/getting-started-java.adoc b/docs/modules/ROOT/pages/tutorials/getting-started-java.adoc index 7463a162..afb71b10 100644 --- a/docs/modules/ROOT/pages/tutorials/getting-started-java.adoc +++ b/docs/modules/ROOT/pages/tutorials/getting-started-java.adoc @@ -25,7 +25,7 @@ In your `build.gradle.kts`: [source,kotlin] ---- dependencies { - implementation(platform("sk.ainet.transformers:skainet-transformers-bom:0.63.3")) + implementation(platform("sk.ainet.transformers:skainet-transformers-bom:0.64.0")) implementation("sk.ainet.transformers:skainet-transformers-runtime-kllama") implementation("sk.ainet.transformers:skainet-transformers-agent") @@ -41,7 +41,7 @@ Or in Maven (Maven needs the `-jvm` classifier suffix on platform artifacts): sk.ainet.transformers skainet-transformers-bom - 0.63.3 + 0.64.0 pom import diff --git a/docs/modules/ROOT/pages/tutorials/getting-started-leaf.adoc b/docs/modules/ROOT/pages/tutorials/getting-started-leaf.adoc index 16119908..90b3b82b 100644 --- a/docs/modules/ROOT/pages/tutorials/getting-started-leaf.adoc +++ b/docs/modules/ROOT/pages/tutorials/getting-started-leaf.adoc @@ -34,7 +34,7 @@ encoder output to the advertised dimensionality. The runtime applies it automati [source,kotlin] ---- dependencies { - implementation(platform("sk.ainet.transformers:skainet-transformers-bom:0.63.3")) + implementation(platform("sk.ainet.transformers:skainet-transformers-bom:0.64.0")) implementation("sk.ainet.transformers:skainet-transformers-providers") } ---- diff --git a/docs/modules/ROOT/pages/tutorials/llama3-tool-calling.adoc b/docs/modules/ROOT/pages/tutorials/llama3-tool-calling.adoc index d44a39bb..9654d325 100644 --- a/docs/modules/ROOT/pages/tutorials/llama3-tool-calling.adoc +++ b/docs/modules/ROOT/pages/tutorials/llama3-tool-calling.adoc @@ -52,7 +52,7 @@ The pieces you need live in three modules: [source,kotlin] ---- dependencies { - implementation(platform("sk.ainet.transformers:skainet-transformers-bom:0.63.3")) + implementation(platform("sk.ainet.transformers:skainet-transformers-bom:0.64.0")) implementation("sk.ainet.transformers:skainet-transformers-runtime-kllama") implementation("sk.ainet.transformers:skainet-transformers-agent") diff --git a/gradle.properties b/gradle.properties index 583f6764..9cff65d6 100644 --- a/gradle.properties +++ b/gradle.properties @@ -1,5 +1,5 @@ GROUP=sk.ainet.transformers -VERSION_NAME=0.63.3 +VERSION_NAME=0.64.0 POM_DESCRIPTION=SKaiNET-transformers diff --git a/llm-inference/lfm2/README.md b/llm-inference/lfm2/README.md index 49844e9d..2978993e 100644 --- a/llm-inference/lfm2/README.md +++ b/llm-inference/lfm2/README.md @@ -1,6 +1,6 @@ # LFM2 model and StableHLO export -This module implements the dense, bias-free LFM2 hybrid architecture using SKaiNET tensor operations. It publishes a JVM target as `sk.ainet.transformers:skainet-transformers-inference-lfm2`; the tensor blocks live in `commonMain`, while checkpoint loading and export tooling run on the JVM. It uses the repository's engine BOM, currently 0.62.0. +This module implements the dense, bias-free LFM2 hybrid architecture using SKaiNET tensor operations. It publishes a JVM target as `sk.ainet.transformers:skainet-transformers-inference-lfm2`; the tensor blocks live in `commonMain`, while checkpoint loading and export tooling run on the JVM. It uses the repository's engine BOM, currently 0.63.0. The supported configuration has short convolution and grouped-query attention layers, split-half RoPE, SwiGLU feed-forward blocks, and a tied embedding/output projection. Unsupported architectural choices fail validation. This does not establish support for every checkpoint marketed as LFM, LFM2, or LFM2.5; MoE, multimodal paths, projection biases, scaled RoPE, quantized execution, and untied heads are outside this implementation.