diff --git a/CHANGELOG.md b/CHANGELOG.md index a0e363e5..fc38e1e4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [0.63.3] — 2026-10-28 + +Against engine **0.62.0**, transformers-only and additive. The Android runtime (`runtime-iree-android`) gains an +external-decoder hook for streaming Moonshine, next to 0.63.2's external encoder; it still depends on nothing but the +Kotlin standard library. + +### Added + +- **Moonshine streaming: external decoder hook** (`moonshine_stream_jni.c`, `libskainet_moonshine_stream.so` both + ABIs). A prefill path of the form `ext:|` replaces the prefill and step graphs with a native + library loaded by `dlopen` that exports `moonshine_ext_decoder()` (ABI 1: `create(dir, vocab, dim, maxtok)`, + `cross(memory, rows)`, `reset`, `step(embedded token, logits)`, `destroy`). The streaming loop, greedy choice, token + budgets, repeat guard and text handling stay in the runtime, and partial transcripts keep their locked prefix + (`cross` refreshes the memory without dropping decoded tokens). The step path is then ignored; with both sides + external no params archive is needed. Without the prefix nothing changes: the IREE step moved into one helper, + behaviour identical (444/444 transcripts identical on the external-encoder + Vulkan-decoder path). Checked on an + arm32 device over 444 German commands with an external encoder and decoder on an accelerator outside IREE: 13.5 % + WER, transcript 238 ms after the end of the audio (p50; 1080 ms with the Vulkan decoder), no GPU use. +- `native/moonshine_stream_check.c`: `MOONSHINE_DEC` replaces the prefill path (e.g. an `ext:` decoder). + ## [0.63.2] — 2026-10-27 Against engine **0.62.0**, transformers-only and additive. The Android runtime (`runtime-iree-android`) gains an diff --git a/README.md b/README.md index 93a07212..35fe723f 100644 --- a/README.md +++ b/README.md @@ -113,7 +113,7 @@ Honest status — see the project-status note at the top of this README. ## Current release -The current release is **0.63.2**, against **SKaiNET 0.62.0**: the streaming Moonshine runtime can hand the encoder side to an external native library (`ext:|` as the encoder path), so an accelerator outside IREE runs it while the decoder stays on the IREE device. 0.63.1 gave the streaming Moonshine runtime an incremental frontend (an optional `frontend_step.vmfb` takes most of the frontend out of the end-of-utterance flush, transcripts unchanged) and `IreeModuleRunner` runs one compiled IREE module on Android. 0.63.0 brought expert heads and routing for Laya — a decision head fine-tuned on top of the frozen encoder (`expert.safetensors` + `expert.json`) loads on its own with `LayaExpertLoader`, any number of heads share one encoder pass per question (`LayaDecisionModel.logitsFrom`), `LayaRouterSpec` + `LayaMixtureAgent` route a request by asking the base head what kind of text it is, and `LayaMixtureDecisionAdapter` puts the mixture behind the typed decision contract; verified against the published layaMOE evaluation. The ModernBERT-large `laya-typed-decisions` checkpoint loads (`position_embedding_type: absolute`). The whisper StableHLO export ships as `WhisperExportCli`. Engine 0.62.0 brings the JVM ↔ Kotlin/Wasm parity: every kernel on the wasm tier, resident activations, fused attention / norms / softmax, browser IO parity. The Laya Q8_0 lane on the desktop JVM with the native provider needs engine 0.62.1 (SKaiNET #1436). +The current release is **0.63.3**, against **SKaiNET 0.62.0**: the streaming Moonshine runtime can hand the decoder to an external native library too (`ext:|` as the prefill path), so with 0.63.2's external encoder the whole model can run on an accelerator outside IREE while the runtime keeps the streaming loop, greedy decoding and its guards. 0.63.2 handed the encoder side to an external library (`ext:` as the encoder path) while the decoder stays on the IREE device. 0.63.1 gave the streaming Moonshine runtime an incremental frontend (an optional `frontend_step.vmfb` takes most of the frontend out of the end-of-utterance flush, transcripts unchanged) and `IreeModuleRunner` runs one compiled IREE module on Android. 0.63.0 brought expert heads and routing for Laya — a decision head fine-tuned on top of the frozen encoder (`expert.safetensors` + `expert.json`) loads on its own with `LayaExpertLoader`, any number of heads share one encoder pass per question (`LayaDecisionModel.logitsFrom`), `LayaRouterSpec` + `LayaMixtureAgent` route a request by asking the base head what kind of text it is, and `LayaMixtureDecisionAdapter` puts the mixture behind the typed decision contract; verified against the published layaMOE evaluation. The ModernBERT-large `laya-typed-decisions` checkpoint loads (`position_embedding_type: absolute`). The whisper StableHLO export ships as `WhisperExportCli`. Engine 0.62.0 brings the JVM ↔ Kotlin/Wasm parity: every kernel on the wasm tier, resident activations, fused attention / norms / softmax, browser IO parity. The Laya Q8_0 lane on the desktop JVM with the native provider needs engine 0.62.1 (SKaiNET #1436). It builds on **0.62.2**, against **SKaiNET 0.61.0**: a transformers-only fix, the Android runtime's streaming Moonshine libraries are rebuilt with the utterance-start fix (#500; 0.62.0 and 0.62.1 packed the old build, which garbles transcripts on device). It builds on **0.62.1**, a transformers-only fix on 0.62.0, whose JVM `LayaModelLoader.load(directory, …)` had become an extension function that existing callers did not import; it is a member again (`@JvmOverloads`, so Java callers keep the 0.61 signature). @@ -303,7 +303,7 @@ The recommended way to consume is via the BOM. It pins every published `skainet- ```kotlin dependencies { - implementation(platform("sk.ainet.transformers:skainet-transformers-bom:0.63.2")) + implementation(platform("sk.ainet.transformers:skainet-transformers-bom:0.63.3")) // Versions resolved from the BOM: implementation("sk.ainet.transformers:skainet-transformers-core") diff --git a/docs/modules/ROOT/pages/how-to/export-moonshine-v2-streaming.adoc b/docs/modules/ROOT/pages/how-to/export-moonshine-v2-streaming.adoc index ed7b0588..a8867080 100644 --- a/docs/modules/ROOT/pages/how-to/export-moonshine-v2-streaming.adoc +++ b/docs/modules/ROOT/pages/how-to/export-moonshine-v2-streaming.adoc @@ -32,7 +32,7 @@ From your own build, resolve the published module and run its main class: ---- val exportTool by configurations.creating dependencies { - exportTool(platform("sk.ainet.transformers:skainet-transformers-bom:0.63.2")) + exportTool(platform("sk.ainet.transformers:skainet-transformers-bom:0.63.3")) exportTool("sk.ainet.transformers:skainet-transformers-inference-moonshine") } tasks.register("exportMoonshine") { diff --git a/docs/modules/ROOT/pages/reference/moonshine-encoder.adoc b/docs/modules/ROOT/pages/reference/moonshine-encoder.adoc index e1196a0f..c65d7f76 100644 --- a/docs/modules/ROOT/pages/reference/moonshine-encoder.adoc +++ b/docs/modules/ROOT/pages/reference/moonshine-encoder.adoc @@ -24,7 +24,7 @@ a DSL decoder is future work. [source,kotlin] ---- dependencies { - implementation(platform("sk.ainet.transformers:skainet-transformers-bom:0.63.2")) + implementation(platform("sk.ainet.transformers:skainet-transformers-bom:0.63.3")) implementation("sk.ainet.transformers:skainet-transformers-inference-moonshine") } ---- diff --git a/docs/modules/ROOT/pages/reference/typed-decisions.adoc b/docs/modules/ROOT/pages/reference/typed-decisions.adoc index 4f449f3f..3dc8a4f8 100644 --- a/docs/modules/ROOT/pages/reference/typed-decisions.adoc +++ b/docs/modules/ROOT/pages/reference/typed-decisions.adoc @@ -12,7 +12,7 @@ below put that on the JVM path with the same contract for every model. [source,kotlin] ---- dependencies { - implementation(platform("sk.ainet.transformers:skainet-transformers-bom:0.63.2")) + implementation(platform("sk.ainet.transformers:skainet-transformers-bom:0.63.3")) implementation("sk.ainet.transformers:skainet-transformers-decisions-api") // the contract, no engine dependency implementation("sk.ainet.transformers:skainet-transformers-decisions-runtime") // validation + acceptance policy implementation("sk.ainet.transformers:skainet-transformers-inference-laya") // Laya as a DecisionAdapter diff --git a/docs/modules/ROOT/pages/tutorials/android-getting-started.adoc b/docs/modules/ROOT/pages/tutorials/android-getting-started.adoc index 1a374ae6..bf28859d 100644 --- a/docs/modules/ROOT/pages/tutorials/android-getting-started.adoc +++ b/docs/modules/ROOT/pages/tutorials/android-getting-started.adoc @@ -55,7 +55,7 @@ dependencies { // self-registers it on ART at process start — nothing to call. runtimeOnly("sk.ainet.core:skainet-backend-jni-cpu") - implementation(platform("sk.ainet.transformers:skainet-transformers-bom:0.63.2")) + implementation(platform("sk.ainet.transformers:skainet-transformers-bom:0.63.3")) implementation("sk.ainet.transformers:skainet-transformers-core") implementation("sk.ainet.transformers:skainet-transformers-runtime-kllama") implementation("sk.ainet.transformers:skainet-transformers-inference-llama") diff --git a/docs/modules/ROOT/pages/tutorials/getting-started-java.adoc b/docs/modules/ROOT/pages/tutorials/getting-started-java.adoc index 7446e8bc..7463a162 100644 --- a/docs/modules/ROOT/pages/tutorials/getting-started-java.adoc +++ b/docs/modules/ROOT/pages/tutorials/getting-started-java.adoc @@ -25,7 +25,7 @@ In your `build.gradle.kts`: [source,kotlin] ---- dependencies { - implementation(platform("sk.ainet.transformers:skainet-transformers-bom:0.63.2")) + implementation(platform("sk.ainet.transformers:skainet-transformers-bom:0.63.3")) implementation("sk.ainet.transformers:skainet-transformers-runtime-kllama") implementation("sk.ainet.transformers:skainet-transformers-agent") @@ -41,7 +41,7 @@ Or in Maven (Maven needs the `-jvm` classifier suffix on platform artifacts): sk.ainet.transformers skainet-transformers-bom - 0.63.2 + 0.63.3 pom import diff --git a/docs/modules/ROOT/pages/tutorials/getting-started-leaf.adoc b/docs/modules/ROOT/pages/tutorials/getting-started-leaf.adoc index befc2b16..16119908 100644 --- a/docs/modules/ROOT/pages/tutorials/getting-started-leaf.adoc +++ b/docs/modules/ROOT/pages/tutorials/getting-started-leaf.adoc @@ -34,7 +34,7 @@ encoder output to the advertised dimensionality. The runtime applies it automati [source,kotlin] ---- dependencies { - implementation(platform("sk.ainet.transformers:skainet-transformers-bom:0.63.2")) + implementation(platform("sk.ainet.transformers:skainet-transformers-bom:0.63.3")) implementation("sk.ainet.transformers:skainet-transformers-providers") } ---- diff --git a/docs/modules/ROOT/pages/tutorials/llama3-tool-calling.adoc b/docs/modules/ROOT/pages/tutorials/llama3-tool-calling.adoc index 5d9d6efd..d44a39bb 100644 --- a/docs/modules/ROOT/pages/tutorials/llama3-tool-calling.adoc +++ b/docs/modules/ROOT/pages/tutorials/llama3-tool-calling.adoc @@ -52,7 +52,7 @@ The pieces you need live in three modules: [source,kotlin] ---- dependencies { - implementation(platform("sk.ainet.transformers:skainet-transformers-bom:0.63.2")) + implementation(platform("sk.ainet.transformers:skainet-transformers-bom:0.63.3")) implementation("sk.ainet.transformers:skainet-transformers-runtime-kllama") implementation("sk.ainet.transformers:skainet-transformers-agent") diff --git a/gradle.properties b/gradle.properties index 660fb55a..583f6764 100644 --- a/gradle.properties +++ b/gradle.properties @@ -1,5 +1,5 @@ GROUP=sk.ainet.transformers -VERSION_NAME=0.63.2 +VERSION_NAME=0.63.3 POM_DESCRIPTION=SKaiNET-transformers diff --git a/llm-runtime/iree-android/native/moonshine_stream_check.c b/llm-runtime/iree-android/native/moonshine_stream_check.c index ef5ccee5..603b6001 100644 --- a/llm-runtime/iree-android/native/moonshine_stream_check.c +++ b/llm-runtime/iree-android/native/moonshine_stream_check.c @@ -12,6 +12,7 @@ * pcm headerless little-endian s16 mono 16 kHz; a directory = every *.pcm in it, name order * --fast feed without pacing (timings then no longer resemble a live stream) * MOONSHINE_ENC (env) replaces the encoder path, e.g. "ext:|" for an external encoder + * MOONSHINE_DEC (env) replaces the prefill path, e.g. "ext:|" for an external decoder * * Build (same image as the library): * docker run --rm -v "$PWD/native":/src -w /src --entrypoint sh skainet/iree-android:3.11.0 -c \ @@ -79,6 +80,7 @@ int main(int argc, char** argv) { static char p[8][1024]; for (int i = 0; i < 8; ++i) snprintf(p[i], sizeof p[i], "%s/%s", argv[3], names[i]); if (getenv("MOONSHINE_ENC")) snprintf(p[1], sizeof p[1], "%s", getenv("MOONSHINE_ENC")); + if (getenv("MOONSHINE_DEC")) snprintf(p[3], sizeof p[3], "%s", getenv("MOONSHINE_DEC")); double t0 = now_ms(); jlong h = create(env, NULL, (jstring)argv[2], (jstring)p[0], (jstring)p[1], (jstring)p[2], (jstring)p[3], (jstring)p[4], (jstring)p[5], (jstring)p[6], (jstring)p[7]); if (!h) { fprintf(stderr, "nativeCreate failed\n"); return 4; } diff --git a/llm-runtime/iree-android/native/moonshine_stream_jni.c b/llm-runtime/iree-android/native/moonshine_stream_jni.c index 1e4c6709..59ac9d3d 100644 --- a/llm-runtime/iree-android/native/moonshine_stream_jni.c +++ b/llm-runtime/iree-android/native/moonshine_stream_jni.c @@ -30,6 +30,11 @@ * `const MoonshineExtEncoder* moonshine_ext_encoder(void)` (struct below): one whole window of audio in, the * window's 64 memory rows (adapter applied) out. It lets an accelerator outside IREE run the encoder side while * the decoder stays on the IREE device. + * + * External decoder: a `prefill` path of the same form "ext:|" replaces the prefill and step + * graphs (step is then ignored; with an external encoder too, no parameter archive is needed). The library exports + * `const MoonshineExtDecoder* moonshine_ext_decoder(void)` (struct below) and decodes one token per step; the + * streaming loop, greedy choice, budgets and guards stay here. * Per-hop stage timings go to stderr/logcat as "moonshine-timing" lines. */ #include @@ -146,6 +151,20 @@ typedef struct { void (*destroy)(void* h); } MoonshineExtEncoder; +/* External decoder ABI, version 1. create(): vocab = rows of the output head, dim = DIM, maxtok = positions the decode + * may reach (NULL when unsupported). cross(): mem[n * DIM] = finalized memory rows (adapter applied) -> the + * cross-attention state; the token state is kept (partials refresh the memory under a locked prefix). reset(): forget + * the tokens. step(): x[DIM] = embedding of the previous token (BOS first) at the next position; logits[vocab] out. + * cross() and step() return 0 on success. */ +typedef struct { + int abi; + void* (*create)(const char* dir, int vocab, int dim, int maxtok); + int (*cross)(void* h, const float* mem, int n); + void (*reset)(void* h); + int (*step)(void* h, const float* x, float* logits); + void (*destroy)(void* h); +} MoonshineExtDecoder; + typedef struct { iree_runtime_instance_t* inst; iree_hal_device_t* dev; @@ -153,6 +172,8 @@ typedef struct { iree_runtime_session_t* fe, *enc, *adp, *pre, *step; iree_runtime_session_t* fes; /* incremental frontend, NULL = whole-window frontend */ void* ext_lib; const MoonshineExtEncoder* ext; void* ext_h; /* external encoder, replaces fe/enc/adp */ + void* dext_lib; const MoonshineExtDecoder* dext; void* dext_h; /* external decoder, replaces pre/step */ + float* logits; /* [vocab.n] external decoder output */ float (*feat)[ENC_DIM]; /* [MAXF] frontend features of this utterance (fes only) */ int fe_done; /* features [0, fe_done) are exact: backed by audio, never recomputed */ double fe_feed_ms; int fe_calls; /* incremental frontend cost while the audio streams in */ @@ -304,12 +325,17 @@ static int pull(Engine* e, iree_hal_buffer_view_t* bv, void* dst, size_t bytes) 0, dst, bytes, IREE_HAL_TRANSFER_BUFFER_FLAG_DEFAULT, iree_infinite_timeout())) ? 0 : -1; } +static int argmax_row(const float* row, int n) { + int best = 0; float b = row[0]; + for (int i = 1; i < n; ++i) if (row[i] > b) { b = row[i]; best = i; } + return best; +} + static int argmax_logits(Engine* e, iree_hal_buffer_view_t* bv) { const int n = e->vocab.n; float* row = malloc((size_t)n * sizeof(float)); if (!row) return EOS; if (pull(e, bv, row, (size_t)n * sizeof(float))) { free(row); return EOS; } - int best = 0; float b = row[0]; - for (int i = 1; i < n; ++i) if (row[i] > b) { b = row[i]; best = i; } + int best = argmax_row(row, n); free(row); return best; } @@ -327,11 +353,13 @@ static void rope_init(Engine* e) { static void release_self(Engine* e) { if (!e->has_self) return; + if (e->dext) { e->dext->reset(e->dext_h); e->has_self = 0; return; } for (int l = 0; l < L; ++l) { iree_hal_buffer_view_release(e->sk[l]); iree_hal_buffer_view_release(e->sv[l]); } e->has_self = 0; } static void release_cross(Engine* e) { if (!e->has_cross) return; + if (e->dext) { e->has_cross = 0; return; } for (int l = 0; l < L; ++l) { iree_hal_buffer_view_release(e->ck[l]); iree_hal_buffer_view_release(e->cv[l]); } e->has_cross = 0; } @@ -554,7 +582,25 @@ static int process_window(Engine* e, int start, int produced, int flush, int pee /* Prefill over the current finalized memory; refresh cross-K/V (and start the token * sequence if none yet). Returns 0 on success. */ +/* run_prefill on the external decoder: refresh the cross state; with start_tokens, restart the tokens and take the + * first one from BOS (the IREE prefill graph's logits). */ +static int run_prefill_ext(Engine* e, int start_tokens) { + if (e->dext->cross(e->dext_h, &e->mem[0][0], e->nmem)) return -1; + e->has_cross = 1; + e->halted = 0; e->ended_eos = 0; + if (!start_tokens) return 0; + e->dext->reset(e->dext_h); + e->has_self = 1; + e->ntoks = 0; e->pos = 0; e->text[0] = 0; + if (e->dext->step(e->dext_h, e->embed + (size_t)BOS * DIM, e->logits)) return -1; + e->pos = 1; + int tok = argmax_row(e->logits, e->vocab.n); + if (tok != EOS) { e->toks[e->ntoks++] = tok; detok_append(&e->vocab, tok, e->text, sizeof(e->text)); } + return 0; +} + static int run_prefill(Engine* e, int start_tokens) { + if (e->dext) return run_prefill_ext(e, start_tokens); static float mem256[MAXMEM][DIM]; static float mask[MAXMEM]; memcpy(mem256, e->mem, sizeof(mem256)); @@ -648,52 +694,64 @@ static void rebuild_text(Engine* e) { for (int i = 0; i < e->ntoks; ++i) detok_append(&e->vocab, e->toks[i], e->text, sizeof(e->text)); } +/* One with_past step on the IREE device: the embedding of `last` at e->pos over the current KV state; + * returns the next token, -1 on error. */ +static int iree_step(Engine* e, int last, const float* mask) { + iree_hal_buffer_view_t* ev = mkview(e, (iree_hal_dim_t[]){1, 1, DIM}, 3, + IREE_HAL_ELEMENT_TYPE_FLOAT_32, e->embed + (size_t)last * DIM, DIM * sizeof(float)); + iree_hal_buffer_view_t* cq = mkview(e, (iree_hal_dim_t[]){1, HD}, 2, + IREE_HAL_ELEMENT_TYPE_FLOAT_32, e->rope_c[e->pos], HD * sizeof(float)); + iree_hal_buffer_view_t* sq = mkview(e, (iree_hal_dim_t[]){1, HD}, 2, + IREE_HAL_ELEMENT_TYPE_FLOAT_32, e->rope_s[e->pos], HD * sizeof(float)); + iree_hal_buffer_view_t* maskv = mkview(e, (iree_hal_dim_t[]){1, 1, 1, MAXMEM}, 4, + IREE_HAL_ELEMENT_TYPE_FLOAT_32, mask, MAXMEM * sizeof(float)); + if (!ev || !cq || !sq || !maskv) return -1; + + iree_runtime_call_t c; + if (!iree_status_is_ok(iree_runtime_call_initialize_by_name( + e->step, iree_make_cstring_view(FN_STEP), &c))) return -1; + iree_runtime_call_inputs_push_back_buffer_view(&c, ev); + iree_runtime_call_inputs_push_back_buffer_view(&c, cq); + iree_runtime_call_inputs_push_back_buffer_view(&c, sq); + for (int l = 0; l < L; ++l) { + iree_runtime_call_inputs_push_back_buffer_view(&c, e->sk[l]); + iree_runtime_call_inputs_push_back_buffer_view(&c, e->sv[l]); + iree_runtime_call_inputs_push_back_buffer_view(&c, e->ck[l]); + iree_runtime_call_inputs_push_back_buffer_view(&c, e->cv[l]); + if (l == 0) iree_runtime_call_inputs_push_back_buffer_view(&c, maskv); + } + int ok = iree_status_is_ok(iree_runtime_call_invoke(&c, 0)); + iree_hal_buffer_view_release(ev); iree_hal_buffer_view_release(cq); + iree_hal_buffer_view_release(sq); iree_hal_buffer_view_release(maskv); + if (!ok) { iree_runtime_call_deinitialize(&c); return -1; } + for (int l = 0; l < L; ++l) { + iree_hal_buffer_view_release(e->sk[l]); iree_hal_buffer_view_release(e->sv[l]); + iree_runtime_call_outputs_pop_front_buffer_view(&c, &e->sk[l]); + iree_runtime_call_outputs_pop_front_buffer_view(&c, &e->sv[l]); + } + iree_hal_buffer_view_t* logits = NULL; + iree_runtime_call_outputs_pop_front_buffer_view(&c, &logits); + iree_runtime_call_deinitialize(&c); + int nxt = argmax_logits(e, logits); + if (logits) iree_hal_buffer_view_release(logits); + return nxt; +} + /* Greedy steps from the current state; stops on EOS / cycle / budget / MAXTOK. `trim` also * drops the detected cycle's repeat from the emitted tokens (safe when the decode state is * rebuilt next hop or discarded at finish). Returns new-token count. */ static int run_steps(Engine* e, int budget, int trim) { - static float mask[MAXMEM]; + static float mask[MAXMEM]; /* IREE decoder only */ for (int f = 0; f < MAXMEM; ++f) mask[f] = f < e->nmem ? 0.0f : NEGMASK; int added = 0; while (e->ntoks > 0 && e->ntoks < MAXTOK && added < budget && e->pos < MAXTOK) { - int last = e->toks[e->ntoks - 1]; - iree_hal_buffer_view_t* ev = mkview(e, (iree_hal_dim_t[]){1, 1, DIM}, 3, - IREE_HAL_ELEMENT_TYPE_FLOAT_32, e->embed + (size_t)last * DIM, DIM * sizeof(float)); - iree_hal_buffer_view_t* cq = mkview(e, (iree_hal_dim_t[]){1, HD}, 2, - IREE_HAL_ELEMENT_TYPE_FLOAT_32, e->rope_c[e->pos], HD * sizeof(float)); - iree_hal_buffer_view_t* sq = mkview(e, (iree_hal_dim_t[]){1, HD}, 2, - IREE_HAL_ELEMENT_TYPE_FLOAT_32, e->rope_s[e->pos], HD * sizeof(float)); - iree_hal_buffer_view_t* maskv = mkview(e, (iree_hal_dim_t[]){1, 1, 1, MAXMEM}, 4, - IREE_HAL_ELEMENT_TYPE_FLOAT_32, mask, sizeof(mask)); - if (!ev || !cq || !sq || !maskv) return added; - - iree_runtime_call_t c; - if (!iree_status_is_ok(iree_runtime_call_initialize_by_name( - e->step, iree_make_cstring_view(FN_STEP), &c))) return added; - iree_runtime_call_inputs_push_back_buffer_view(&c, ev); - iree_runtime_call_inputs_push_back_buffer_view(&c, cq); - iree_runtime_call_inputs_push_back_buffer_view(&c, sq); - for (int l = 0; l < L; ++l) { - iree_runtime_call_inputs_push_back_buffer_view(&c, e->sk[l]); - iree_runtime_call_inputs_push_back_buffer_view(&c, e->sv[l]); - iree_runtime_call_inputs_push_back_buffer_view(&c, e->ck[l]); - iree_runtime_call_inputs_push_back_buffer_view(&c, e->cv[l]); - if (l == 0) iree_runtime_call_inputs_push_back_buffer_view(&c, maskv); - } - int ok = iree_status_is_ok(iree_runtime_call_invoke(&c, 0)); - iree_hal_buffer_view_release(ev); iree_hal_buffer_view_release(cq); - iree_hal_buffer_view_release(sq); iree_hal_buffer_view_release(maskv); - if (!ok) { iree_runtime_call_deinitialize(&c); return added; } - for (int l = 0; l < L; ++l) { - iree_hal_buffer_view_release(e->sk[l]); iree_hal_buffer_view_release(e->sv[l]); - iree_runtime_call_outputs_pop_front_buffer_view(&c, &e->sk[l]); - iree_runtime_call_outputs_pop_front_buffer_view(&c, &e->sv[l]); + int last = e->toks[e->ntoks - 1], nxt; + if (e->dext) { + if (e->dext->step(e->dext_h, e->embed + (size_t)last * DIM, e->logits)) return added; + nxt = argmax_row(e->logits, e->vocab.n); + } else if ((nxt = iree_step(e, last, mask)) < 0) { + return added; } - iree_hal_buffer_view_t* logits = NULL; - iree_runtime_call_outputs_pop_front_buffer_view(&c, &logits); - iree_runtime_call_deinitialize(&c); - int nxt = argmax_logits(e, logits); - if (logits) iree_hal_buffer_view_release(logits); e->pos++; if (nxt == EOS) { e->halted = 1; e->ended_eos = 1; break; } /* tentative while streaming */ e->toks[e->ntoks++] = nxt; @@ -782,24 +840,45 @@ static void reset_utterance(Engine* e) { e->ntoks = 0; e->pos = 0; e->text[0] = 0; } -/* "|": dlopen the external encoder and create it over the model dir. */ -static void load_ext(Engine* e, const char* spec) { - char lib[1024]; +/* "|": dlopen the library and return its `sym()` table (NULL on failure); *dir = the model dir. */ +static const void* open_ext(const char* spec, const char* sym, void** lib, const char** dir) { + char path[1024]; const char* bar = strchr(spec, '|'); - if (!bar || (size_t)(bar - spec) >= sizeof(lib)) { TLOG("create: ext(%s) -> FAIL (want lib|dir)\n", spec); return; } - memcpy(lib, spec, bar - spec); lib[bar - spec] = 0; - e->ext_lib = dlopen(lib, RTLD_NOW | RTLD_LOCAL); - const MoonshineExtEncoder* (*get)(void) = e->ext_lib ? (const MoonshineExtEncoder* (*)(void))dlsym(e->ext_lib, "moonshine_ext_encoder") : NULL; - const MoonshineExtEncoder* x = get ? get() : NULL; - if (x && x->abi == 1) e->ext_h = x->create(bar + 1); + *lib = NULL; + if (!bar || (size_t)(bar - spec) >= sizeof(path)) { TLOG("create: ext(%s) -> FAIL (want lib|dir)\n", spec); return NULL; } + memcpy(path, spec, bar - spec); path[bar - spec] = 0; + *dir = bar + 1; + *lib = dlopen(path, RTLD_NOW | RTLD_LOCAL); + if (!*lib) { TLOG("create: ext(%s) -> FAIL %s\n", spec, dlerror()); return NULL; } + const void* (*get)(void) = (const void* (*)(void))dlsym(*lib, sym); + return get ? get() : NULL; +} + +static void load_ext(Engine* e, const char* spec) { + const char* dir; + const MoonshineExtEncoder* x = open_ext(spec, "moonshine_ext_encoder", &e->ext_lib, &dir); + if (x && x->abi == 1) e->ext_h = x->create(dir); if (e->ext_h) e->ext = x; - TLOG("create: ext(%s) -> %s%s\n", spec, e->ext ? "ok" : "FAIL ", e->ext ? "" : (e->ext_lib ? "(symbol/abi/create)" : dlerror())); + TLOG("create: ext(%s) -> %s\n", spec, e->ext ? "ok" : "FAIL (symbol/abi/create)"); +} + +static void load_dext(Engine* e, const char* spec) { + const char* dir; + const MoonshineExtDecoder* x = open_ext(spec, "moonshine_ext_decoder", &e->dext_lib, &dir); + e->logits = x ? malloc((size_t)e->vocab.n * sizeof(float)) : NULL; + if (x && x->abi == 1 && e->logits) e->dext_h = x->create(dir, e->vocab.n, DIM, MAXTOK + 1); + if (e->dext_h) e->dext = x; + TLOG("create: dext(%s) -> %s\n", spec, e->dext ? "ok" : "FAIL (symbol/abi/create)"); } static void release_ext(Engine* e) { if (e->ext && e->ext_h) e->ext->destroy(e->ext_h); if (e->ext_lib) dlclose(e->ext_lib); e->ext = NULL; e->ext_h = NULL; e->ext_lib = NULL; + if (e->dext && e->dext_h) e->dext->destroy(e->dext_h); + if (e->dext_lib) dlclose(e->dext_lib); + e->dext = NULL; e->dext_h = NULL; e->dext_lib = NULL; + free(e->logits); e->logits = NULL; } /* ---- JNI -------------------------------------------------------------------------------- */ @@ -826,9 +905,12 @@ JNIEXPORT jlong JNICALL JNIFN(nativeCreate)(JNIEnv* env, jobject thiz, iree_status_is_ok(iree_runtime_instance_create(&io, iree_allocator_system(), &e->inst)) && iree_status_is_ok(iree_runtime_instance_try_create_default_device(e->inst, iree_make_cstring_view(dev), &e->dev))) { - e->params = load_params(e, par); - TLOG("create: params(%s) -> %s\n", par, e->params ? "ok" : "FAIL"); - const int ext = !strncmp(enc, "ext:", 4); + const int ext = !strncmp(enc, "ext:", 4), dext = !strncmp(pre, "ext:", 4); + /* the parameter archive feeds the IREE decoder graphs only */ + if (!(ext && dext)) { + e->params = load_params(e, par); + TLOG("create: params(%s) -> %s\n", par, e->params ? "ok" : "FAIL"); + } if (ext) load_ext(e, enc + 4); else { e->fe = load_session(e, fe, 0); @@ -853,13 +935,17 @@ JNIEXPORT jlong JNICALL JNIFN(nativeCreate)(JNIEnv* env, jobject thiz, e->adp = load_session(e, adp, 0); TLOG("create: adp(%s) -> %s\n", adp, e->adp ? "ok" : "FAIL"); } - if ((ext ? (void*)e->ext : (void*)e->fe) && e->params) { - e->pre = load_session(e, pre, 1); - TLOG("create: pre(%s) -> %s\n", pre, e->pre ? "ok" : "FAIL"); - e->step = load_session(e, stp, 1); - TLOG("create: step(%s) -> %s\n", stp, e->step ? "ok" : "FAIL"); + if ((ext ? (void*)e->ext : (void*)e->fe) && (e->params || (ext && dext))) { e->vocab = vocab_load(voc); TLOG("create: vocab(%s) -> n=%d\n", voc, e->vocab.n); + if (dext) { + if (e->vocab.n > 0) load_dext(e, pre + 4); + } else { + e->pre = load_session(e, pre, 1); + TLOG("create: pre(%s) -> %s\n", pre, e->pre ? "ok" : "FAIL"); + e->step = load_session(e, stp, 1); + TLOG("create: step(%s) -> %s\n", stp, e->step ? "ok" : "FAIL"); + } FILE* f = e->vocab.n > 0 ? fopen(emb, "rb") : NULL; if (f) { const size_t want = (size_t)e->vocab.n * DIM; @@ -874,7 +960,7 @@ JNIEXPORT jlong JNICALL JNIFN(nativeCreate)(JNIEnv* env, jobject thiz, e->pcm = malloc(MAXPCM * sizeof(float)); rope_init(e); if (!e->alloc && e->pre) e->alloc = iree_runtime_session_device_allocator(e->pre); - if ((e->ext || (e->enc && e->adp)) && e->pre && e->step && e->vocab.n > 0 && e->embed && e->pcm) { + if ((e->ext || (e->enc && e->adp)) && (e->dext || (e->pre && e->step)) && e->vocab.n > 0 && e->embed && e->pcm) { warmup(e); ret = (jlong)(intptr_t)e; } diff --git a/llm-runtime/iree-android/src/main/jniLibs/arm64-v8a/libskainet_moonshine_stream.so b/llm-runtime/iree-android/src/main/jniLibs/arm64-v8a/libskainet_moonshine_stream.so index 0f819b13..0f9b60c9 100755 Binary files a/llm-runtime/iree-android/src/main/jniLibs/arm64-v8a/libskainet_moonshine_stream.so and b/llm-runtime/iree-android/src/main/jniLibs/arm64-v8a/libskainet_moonshine_stream.so differ diff --git a/llm-runtime/iree-android/src/main/jniLibs/armeabi-v7a/libskainet_moonshine_stream.so b/llm-runtime/iree-android/src/main/jniLibs/armeabi-v7a/libskainet_moonshine_stream.so index 8c214130..143551b3 100755 Binary files a/llm-runtime/iree-android/src/main/jniLibs/armeabi-v7a/libskainet_moonshine_stream.so and b/llm-runtime/iree-android/src/main/jniLibs/armeabi-v7a/libskainet_moonshine_stream.so differ diff --git a/llm-runtime/iree-android/src/main/kotlin/sk/ainet/transformers/iree/android/IreeMoonshineStream.kt b/llm-runtime/iree-android/src/main/kotlin/sk/ainet/transformers/iree/android/IreeMoonshineStream.kt index df0206ed..d24c49d5 100644 --- a/llm-runtime/iree-android/src/main/kotlin/sk/ainet/transformers/iree/android/IreeMoonshineStream.kt +++ b/llm-runtime/iree-android/src/main/kotlin/sk/ainet/transformers/iree/android/IreeMoonshineStream.kt @@ -21,6 +21,12 @@ package sk.ainet.transformers.iree.android * with dlopen — a bare library name resolves among the app's own native libraries. The decoder stays on [device]; * frontend and adapter paths are then ignored. * + * External decoder: a [Files.prefillVmfb] of the same form, `ext:|`, replaces prefill and step + * with a library exporting `moonshine_ext_decoder()` (ABI 1: cross-attention state from the memory rows, then one + * token per step from the embedded previous token, logits out). The streaming loop, greedy choice, token budgets, + * repeat guard and text handling stay in the runtime; partial transcripts keep their locked prefix. The step path is + * then ignored, and with both sides external the params archive is not needed (pass an empty path). + * * One instance = one utterance at a time; callers serialize access. Per-hop stage timings are logged * under the `moonshine-timing` tag. *