From 12ca52d446089a9765281da48aa7eded1a4c2871 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 9 Oct 2026 19:13:19 +0000 Subject: [PATCH 1/8] Upgrade llama.cpp from b11529 to b11530 All eleven patches apply unchanged (verified in order against the pristine tag); see docs/history/llama-cpp-breaking-changes.md for the range. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01AytmJF9faEiQEVt6eetQS2 --- CLAUDE.md | 8 ++++---- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 ++ llama/CMakeLists.txt | 2 +- .../java/net/ladenthin/llama/value/LlamaCppVersion.java | 8 ++++---- 5 files changed, 12 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 1a814b312..fe06dd745 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b11529** +Current llama.cpp pinned version: **b11530** ## Natives jars: one directory per backend (`.github/natives.csv`) @@ -897,7 +897,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network for the asset build; the embed step is plain cmake -P -git clone --depth 1 --branch b11529 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b11530 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build ) mkdir -p webui-generated /tmp/ui-gen cmake -DUI_SOURCE_DIR=/tmp/lc/tools/ui -DUI_BINARY_DIR=/tmp/ui-gen \ @@ -937,7 +937,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b11529`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b11530`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -2098,7 +2098,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b11529`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b11530`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.18.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index f49bf3039..b52dc3194 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b11529](https://img.shields.io/badge/llama.cpp-%23b11529-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b11529) +[![llama.cpp b11530](https://img.shields.io/badge/llama.cpp-%23b11530-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b11530) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index b40596f3f..f22f62a26 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -860,3 +860,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b11518–b11521 | patches + upstream verification | **All eleven patches apply unchanged**, verified in order against pristine b11521 with `git apply`. Patch-target files in the range: `common/arg.cpp` (`0001`, `0015`: one help string of `--server-base`, away from `common_params_parse` and `add_rpc_devices`) and `tools/server/server.cpp` (`0006`, `0007`: the eight removed lines sat after the `listening on` loop of `llama_server()`, between `0006`'s embedded-mode parse hunk and the router branch, and outside the route table `0007` factors out -- both hunks apply with their context intact). Drop-checks unchanged: `common_params_parse` still carries the Windows `argv` override, the load-progress assignment is still unconditional, no embedded-mode or attach counterpart upstream. | | b11521–b11529 | Eight commits, 22 files, ~311/187 lines (49 KiB, the final step). **ggml: `ggml_acc` out-of-bounds write with a large offset** (#30135, `ggml/src/ggml.c`): the `size_t` offset was narrowed to `int32_t` unchecked, a value past `INT32_MAX` became negative, the forward sign-extended it and the bounds assertion wrapped, so the write landed below the destination buffer; the offset is now checked before the narrowing, as `ggml_set_impl` already did -- core code every natives jar builds. **Vulkan: `rms_norm` workgroup-count overflow** (#30145; `ggml-vulkan.cpp`, `rms_norm.comp`) for the three Vulkan jars. **Meta backend handles host views** (#30217): `--split-mode tensor` with partial offload asserted on KV-cache views left inside a meta split; such nodes are accepted when they are views of host tensors, and `LLM_ARCH_K2_HORIZON` leaves the list of architectures `llm_arch_supports_sm_tensor()` refuses (`src/llama-arch.cpp`, one line) -- relevant to `GpuSplitMode.TENSOR`. MUSA drops `mp_21` from its default architectures (#30203), WebGPU gets 2D workgroup dispatch and `GGML_OP_DUP` (#30219, #30216), the converter reads compressed-tensor mixed-precision NVFP4 checkpoints (#28636, Python), upstream's publish workflow gets a run name (#30214) -- none of it built or run here. **No project source change.** | | b11521–b11529 | patches + upstream verification | **All eleven patches apply unchanged**, verified in order against pristine b11529 with `git apply` and by a fresh configure through the real `FetchContent` path. Patch-target file in the range: `src/llama-arch.cpp` (`0016`: the Kolibri-1 entries sit in the arch and tensor tables, the removed line in `llm_arch_supports_sm_tensor()`). Drop-checks against pristine b11529 -- the final state of the bump: `common_params_parse` still carries the `#ifdef _WIN32` `argv = utf8.ptrs.data()` override and `common_params_parse_main` is absent (`0001`); the load-progress assignment is unconditional (`0002`); no `slot_prompt_similarity` getter (`0003`); no embedded-mode, attach or worker-command counterpart (`0006`-`0008`); no `split_sum == 0` guard (`0012`); `common/log.h` names only `common_log_default_callback`, no sink (`0014`); no `stop_server` and the connect failure still aborts in `ggml-rpc.cpp` (`0015`); no `kolibri` under `src/` (`0016`); the four `_mm_prefetch` calls in `quants.c` are still uncast element arithmetic (`0017`). No `.github/` change upstream that concerns this project (only its own `release-publish.yml`). | +| b11529–b11530 | One commit, 2 files, ~16/29 lines (5 KiB, one step). **#30223 keeps the backend sampling graph static across ubatches** (`src/llama-context.cpp`, `src/llama-graph.cpp`): the reserve built `n_outputs_max_per_seq` sampling chains per sampler while a decode built one per output row, so the topology changed after the reserve and `GGML_SCHED_NO_REALLOC` builds aborted on the next same-sized graph; every sampler now builds the full set, the chains without a row land on the padding row, and `graph_max_nodes` counts them. Backend sampling (`setDraftSampling`, the `--spec-draft-sampling` path) is the affected feature; nothing on the wire changes. **No project source change.** | +| b11529–b11530 | patches + upstream verification | **All eleven patches apply unchanged**, verified in order against pristine b11530 with `git apply`. No patch-target file in the range (`0012` and `0016` touch `src/llama-model.cpp`, not the two files here), so every drop-check of the b11521–b11529 row holds unchanged. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index ea36b9d42..6dda90d6f 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -261,7 +261,7 @@ endif() FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b11529 + GIT_TAG b11530 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index 550fb48b2..9c5fd4d57 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -9,13 +9,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b11529"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b11530"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b11529-"} — call + * plus the resolved upstream commit, e.g. {@code "b11530-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -23,14 +23,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b11529"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b11530"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b11529"; + public static final String LLAMA_CPP_VERSION = "b11530"; // Constants holder — not instantiable. private LlamaCppVersion() {} From 4ecb8380fb2bd78681764a916b3f310cccfb61fd Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 9 Oct 2026 19:21:45 +0000 Subject: [PATCH 2/8] Upgrade llama.cpp from b11530 to b11531 Upstream #30210 ("chat : refactor API") moves the chat prompt and parser state into a common_chat_session: oaicompat_chat_params_parse() takes the session as its last argument, task_params carries chat_format + reasoning_format instead of chat_parser_params, server_task::create_state() is gone and server_response_reader::post_tasks() builds the result state from the session. jllama.cpp threads a session through the chat entry points (applyTemplate, handleChatCompletions, requestChatCompletion, requestChatCompletionStream) and applies it to the task before posting; the C++ tests follow the new signatures (603 tests, unchanged count). All eleven patches apply unchanged (verified in order against the pristine tag); see docs/history/llama-cpp-breaking-changes.md for the range. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01AytmJF9faEiQEVt6eetQS2 --- CLAUDE.md | 8 +-- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 + llama/CMakeLists.txt | 2 +- llama/src/main/cpp/jllama.cpp | 53 +++++++++++++------ .../llama/value/LlamaCppVersion.java | 8 +-- llama/src/test/cpp/test_server.cpp | 31 ++++++----- llama/src/test/cpp/test_utils.cpp | 34 +++++++----- 8 files changed, 87 insertions(+), 53 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index fe06dd745..b3261b7dd 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b11530** +Current llama.cpp pinned version: **b11531** ## Natives jars: one directory per backend (`.github/natives.csv`) @@ -897,7 +897,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network for the asset build; the embed step is plain cmake -P -git clone --depth 1 --branch b11530 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b11531 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build ) mkdir -p webui-generated /tmp/ui-gen cmake -DUI_SOURCE_DIR=/tmp/lc/tools/ui -DUI_BINARY_DIR=/tmp/ui-gen \ @@ -937,7 +937,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b11530`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b11531`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -2098,7 +2098,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b11530`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b11531`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.18.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index b52dc3194..9c888c90c 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b11530](https://img.shields.io/badge/llama.cpp-%23b11530-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b11530) +[![llama.cpp b11531](https://img.shields.io/badge/llama.cpp-%23b11531-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b11531) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index f22f62a26..38c732e98 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -862,3 +862,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b11521–b11529 | patches + upstream verification | **All eleven patches apply unchanged**, verified in order against pristine b11529 with `git apply` and by a fresh configure through the real `FetchContent` path. Patch-target file in the range: `src/llama-arch.cpp` (`0016`: the Kolibri-1 entries sit in the arch and tensor tables, the removed line in `llm_arch_supports_sm_tensor()`). Drop-checks against pristine b11529 -- the final state of the bump: `common_params_parse` still carries the `#ifdef _WIN32` `argv = utf8.ptrs.data()` override and `common_params_parse_main` is absent (`0001`); the load-progress assignment is unconditional (`0002`); no `slot_prompt_similarity` getter (`0003`); no embedded-mode, attach or worker-command counterpart (`0006`-`0008`); no `split_sum == 0` guard (`0012`); `common/log.h` names only `common_log_default_callback`, no sink (`0014`); no `stop_server` and the connect failure still aborts in `ggml-rpc.cpp` (`0015`); no `kolibri` under `src/` (`0016`); the four `_mm_prefetch` calls in `quants.c` are still uncast element arithmetic (`0017`). No `.github/` change upstream that concerns this project (only its own `release-publish.yml`). | | b11529–b11530 | One commit, 2 files, ~16/29 lines (5 KiB, one step). **#30223 keeps the backend sampling graph static across ubatches** (`src/llama-context.cpp`, `src/llama-graph.cpp`): the reserve built `n_outputs_max_per_seq` sampling chains per sampler while a decode built one per output row, so the topology changed after the reserve and `GGML_SCHED_NO_REALLOC` builds aborted on the next same-sized graph; every sampler now builds the full set, the chains without a row land on the padding row, and `graph_max_nodes` counts them. Backend sampling (`setDraftSampling`, the `--spec-draft-sampling` path) is the affected feature; nothing on the wire changes. **No project source change.** | | b11529–b11530 | patches + upstream verification | **All eleven patches apply unchanged**, verified in order against pristine b11530 with `git apply`. No patch-target file in the range (`0012` and `0016` touch `src/llama-model.cpp`, not the two files here), so every drop-check of the b11521–b11529 row holds unchanged. | +| b11530–b11531 | One commit, 41 files, ~562/736 lines (119 KiB, forced single step: the one tag between b11530 and b11532). **#30210 refactors the chat API around `common_chat_session`** (`common/chat.{h,cpp}`): one object per generation that builds the prompt and owns the parser state (`feed()`/`finish()`), replacing the parser parameters that used to travel as request fields. Consequences on the server side, each of which reaches this project: `oaicompat_chat_params_parse()` takes the vocab and an `out_session` and no longer writes `chat_format`, `chat_parser`, `generation_prompt` or `parse_tool_calls` into the request JSON; `server_schema` drops those four fields (`echo` stays, parsed for validation only) -- none of them was a `RequestField` constant, so the Java wire surface is unchanged and `test_wire_contracts` keeps passing; `task_params::chat_parser_params` becomes `chat_format` + `reasoning_format` (`reasoning_in_content` in `to_json()` is now derived from `stream` and the legacy DeepSeek format, `generation_prompt` from `sampling.generation_prompt`); `server_task::create_state()` is gone and `server_response_reader::post_tasks()` takes the session, from which it builds every `task_result_state`; `server_task::apply_chat_session()` applies the session's sampling, stops and message spans. `common_chat_template` now picks its dedicated handler or runs the differential autoparser once per template (`params_init`/`analysis`), `common_chat_msg_delimiters::to_json()`/`_parse()` and `peg-parser`'s JSON serialisation are removed, and `common_sampling_add_preserved_tokens` / `common_sampling_add_grammar_triggers` move the tokenisation out of the schema. **Project source changes:** `jllama.cpp`'s `parse_oai_chat_params()` returns the session, the two dispatchers call `apply_chat_session()` and post through the session-taking `post_tasks()` (a default session for the non-chat paths, where it is a no-op); `test_utils.cpp`'s eleven early-validation tests call the new signature through one helper (the throws still fire before the session is built, so `tmpls` and the vocab stay null); `test_server.cpp`'s `task_params::to_json()` tests set `sampling.generation_prompt` and `stream` + `reasoning_format` instead of the removed members. Request-field diff (the mechanical check): `-chat_format -chat_parser -generation_prompt -parse_tool_calls`, `echo` from `field_bool` to `field_json`; response keys of `server-task.cpp` identical. | +| b11530–b11531 | patches + upstream verification | **All eleven patches apply unchanged**, verified in order against pristine b11531 with `git apply` and by a fresh configure, build and `ctest` (603/603) through the real `FetchContent` path, since this step changes project code. Patch-target files in the range: `tools/server/server-context.cpp` and `server-context.h` (`0002`, `0003`: the session plumbing sits in `handle_completions_impl` and the route lambdas, away from `load_model` and the similarity getter/setter). Drop-checks unchanged. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index 6dda90d6f..a7f34eaad 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -261,7 +261,7 @@ endif() FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b11530 + GIT_TAG b11531 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/cpp/jllama.cpp b/llama/src/main/cpp/jllama.cpp index 77aa5930b..1dd69fd8f 100644 --- a/llama/src/main/cpp/jllama.cpp +++ b/llama/src/main/cpp/jllama.cpp @@ -293,14 +293,17 @@ static void wake_server(jllama_context *jctx) { /** * Parse the OAI chat-completion body through oaicompat_chat_params_parse and - * write the result into `out`, preserving decoded media in `files`. Returns - * true on success; on failure throws and returns false. + * write the result into `out`, preserving decoded media in `files` and the chat + * session (prompt, parser state, sampling additions; llama.cpp b11531, #30210) in + * `session`. The session must reach the task (`apply_chat_session`) and the + * reader (`post_tasks`), which the two dispatchers below do. Returns true on + * success; on failure throws and returns false. */ [[nodiscard]] static bool parse_oai_chat_params(JNIEnv *env, jllama_context *jctx, json &body, json &out, - std::vector &files) { + std::vector &files, common_chat_session &session) { try { auto meta = wake_and_get_meta(jctx); - out = oaicompat_chat_params_parse(body, meta.chat_params, files); + out = oaicompat_chat_params_parse(jctx->vocab, body, meta.chat_params, files, session); return true; } catch (const std::exception &e) { throw_invalid_request(env, e); @@ -354,14 +357,19 @@ static void wake_and_post(server_response_reader &rd, server_task &&task, bool f rd.post_task(std::move(task), front); } -static void wake_and_post(server_response_reader &rd, std::vector &&tasks) { +// The chat session (if any) is what the reader builds its task_result_state from -- the +// parser that turns generated text into content, reasoning and tool calls. A default session +// is the non-chat case (content only). +static void wake_and_post(server_response_reader &rd, std::vector &&tasks, + const common_chat_session &session = {}) { rd.queue_tasks.wait_until_no_sleep(); - rd.post_tasks(std::move(tasks)); + rd.post_tasks(std::move(tasks), session); } [[nodiscard]] static jint dispatch_streaming_completion(JNIEnv *env, jllama_context *jctx, const json &data, server_task_type task_type, task_response_type res_type, - std::vector files = {}) { + std::vector files = {}, + const common_chat_session &session = {}) { server_context *ctx_server = &jctx->server; auto meta = wake_and_get_meta(jctx); auto *rd = new server_response_reader(ctx_server->get_response_reader()); @@ -370,8 +378,11 @@ static void wake_and_post(server_response_reader &rd, std::vector & server_task task(task_type); task.id = tid; populate_completion_task(task, jctx, meta.logit_bias_eog, data, meta.has_mtmd, std::move(files)); + task.apply_chat_session(session); task.params.res_type = res_type; - wake_and_post(*rd, std::move(task)); + std::vector tasks; + tasks.push_back(std::move(task)); + wake_and_post(*rd, std::move(tasks), session); } catch (const std::exception &e) { delete rd; throw_invalid_request(env, e); @@ -391,7 +402,8 @@ static void wake_and_post(server_response_reader &rd, std::vector & */ [[nodiscard]] static jstring dispatch_blocking_completion(JNIEnv *env, jllama_context *jctx, const json &data, server_task_type task_type, task_response_type res_type, - std::vector files = {}) { + std::vector files = {}, + const common_chat_session &session = {}) { server_context *ctx_server = &jctx->server; auto meta = wake_and_get_meta(jctx); auto rd = ctx_server->get_response_reader(); @@ -403,8 +415,11 @@ static void wake_and_post(server_response_reader &rd, std::vector & throw_invalid_request(env, e); return nullptr; } + task.apply_chat_session(session); task.params.res_type = res_type; - wake_and_post(rd, std::move(task)); + std::vector tasks; + tasks.push_back(std::move(task)); + wake_and_post(rd, std::move(tasks), session); auto br = rd.wait_for_all([jctx] { return jctx->closing.load(); }); if (!batch_ok_or_throw(env, br)) return nullptr; @@ -1509,7 +1524,8 @@ JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_applyTemplate(JNIE json templateData; std::vector files; - if (!parse_oai_chat_params(env, jctx, data, templateData, files)) + common_chat_session session; // built with the prompt, not needed afterwards + if (!parse_oai_chat_params(env, jctx, data, templateData, files, session)) return nullptr; if (!templateData.contains("prompt") || !templateData.at("prompt").is_string()) { @@ -1532,11 +1548,12 @@ JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_handleChatCompleti } json data; std::vector files; - if (!parse_oai_chat_params(env, jctx, body, data, files)) + common_chat_session session; + if (!parse_oai_chat_params(env, jctx, body, data, files, session)) return nullptr; return dispatch_blocking_completion(env, jctx, data, SERVER_TASK_TYPE_COMPLETION, TASK_RESPONSE_TYPE_OAI_CHAT, - std::move(files)); + std::move(files), session); }); } @@ -1552,11 +1569,12 @@ JNIEXPORT jint JNICALL Java_net_ladenthin_llama_LlamaModel_requestChatCompletion // Chat template already applied by parse_oai_chat_params; no OAI wrapping on the streaming path. json data; std::vector files; - if (!parse_oai_chat_params(env, jctx, body, data, files)) + common_chat_session session; + if (!parse_oai_chat_params(env, jctx, body, data, files, session)) return 0; return dispatch_streaming_completion(env, jctx, data, SERVER_TASK_TYPE_COMPLETION, TASK_RESPONSE_TYPE_NONE, - std::move(files)); + std::move(files), session); }); } @@ -1576,11 +1594,12 @@ JNIEXPORT jint JNICALL Java_net_ladenthin_llama_LlamaModel_requestChatCompletion } json data; std::vector files; - if (!parse_oai_chat_params(env, jctx, body, data, files)) + common_chat_session session; + if (!parse_oai_chat_params(env, jctx, body, data, files, session)) return 0; return dispatch_streaming_completion(env, jctx, data, SERVER_TASK_TYPE_COMPLETION, TASK_RESPONSE_TYPE_OAI_CHAT, - std::move(files)); + std::move(files), session); }); } diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index 9c5fd4d57..0ebca1f49 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -9,13 +9,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b11530"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b11531"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b11530-"} — call + * plus the resolved upstream commit, e.g. {@code "b11531-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -23,14 +23,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b11530"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b11531"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b11530"; + public static final String LLAMA_CPP_VERSION = "b11531"; // Constants holder — not instantiable. private LlamaCppVersion() {} diff --git a/llama/src/test/cpp/test_server.cpp b/llama/src/test/cpp/test_server.cpp index 5a194c325..1615ca29d 100644 --- a/llama/src/test/cpp/test_server.cpp +++ b/llama/src/test/cpp/test_server.cpp @@ -177,6 +177,10 @@ TEST(ServerSlotStats, DraftFieldsAbsent_WhenExplicitlyZero) { // reasoning_format from oaicompat_chat_syntax.reasoning_format // reasoning_in_content from oaicompat_chat_syntax.reasoning_in_content // generation_prompt from oaicompat_chat_syntax.generation_prompt +// Since b11531 (#30210) the parser state lives in common_chat_session, and +// task_params keeps only chat_format + reasoning_format: reasoning_in_content +// is derived (stream && DEEPSEEK_LEGACY), generation_prompt comes from +// sampling.generation_prompt. // ============================================================ TEST(SlotParamsToJson, CoreFields_Present) { @@ -201,13 +205,11 @@ TEST(SlotParamsToJson, NewChatSyntaxFields_Present) { task_params p; const json j = p.to_json(); - EXPECT_TRUE(j.contains("chat_format")) << "chat_format must come from oaicompat_chat_syntax.format"; - EXPECT_TRUE(j.contains("reasoning_format")) - << "reasoning_format must come from oaicompat_chat_syntax.reasoning_format"; + EXPECT_TRUE(j.contains("chat_format")) << "chat_format must come from task_params::chat_format"; + EXPECT_TRUE(j.contains("reasoning_format")) << "reasoning_format must come from task_params::reasoning_format"; EXPECT_TRUE(j.contains("reasoning_in_content")) - << "reasoning_in_content must come from oaicompat_chat_syntax.reasoning_in_content"; - EXPECT_TRUE(j.contains("generation_prompt")) - << "generation_prompt must come from oaicompat_chat_syntax.generation_prompt"; + << "reasoning_in_content must be derived from stream + reasoning_format (b11531)"; + EXPECT_TRUE(j.contains("generation_prompt")) << "generation_prompt must come from sampling.generation_prompt"; } TEST(SlotParamsToJson, OldChatFormatEnum_NotPresent) { @@ -248,22 +250,25 @@ TEST(SlotParamsToJson, GrammarValue_OutputFormatGrammarExtracted) { EXPECT_EQ(j.at("grammar").get(), "root ::= object"); } -TEST(SlotParamsToJson, GenerationPrompt_ReflectsSyntaxField) { +TEST(SlotParamsToJson, GenerationPrompt_ReflectsSamplingField) { task_params p; - p.chat_parser_params.generation_prompt = common_chat_input("Think step by step:"); + p.sampling.generation_prompt = "Think step by step:"; const json j = p.to_json(); EXPECT_EQ(j.at("generation_prompt").get(), "Think step by step:"); } -TEST(SlotParamsToJson, ReasoningInContent_ReflectsSyntaxField) { +TEST(SlotParamsToJson, ReasoningInContent_DerivedFromStreamAndLegacyFormat) { + // b11531: no longer a stored flag -- true exactly for a streamed request in the legacy + // DeepSeek reasoning format, the one case upstream inlines reasoning into the content. task_params p; - p.chat_parser_params.reasoning_in_content = true; + p.stream = true; + p.reasoning_format = COMMON_REASONING_FORMAT_DEEPSEEK_LEGACY; + EXPECT_TRUE(p.to_json().at("reasoning_in_content").get()); - const json j = p.to_json(); - - EXPECT_TRUE(j.at("reasoning_in_content").get()); + p.stream = false; + EXPECT_FALSE(p.to_json().at("reasoning_in_content").get()) << "non-streamed: never inlined"; } TEST(SlotParamsToJson, ReasoningInContent_FalseByDefault) { diff --git a/llama/src/test/cpp/test_utils.cpp b/llama/src/test/cpp/test_utils.cpp index 4cda471a7..54ff316ab 100644 --- a/llama/src/test/cpp/test_utils.cpp +++ b/llama/src/test/cpp/test_utils.cpp @@ -996,14 +996,22 @@ TEST(SafeJsonToStr, InvalidUtf8InString_DoesNotThrow) { // ============================================================ // oaicompat_chat_params_parse — early validation throws -// These all fire BEFORE common_chat_templates_apply is called, -// so opt.tmpls can remain nullptr safely. +// These all fire BEFORE the chat session is built (llama.cpp b11531, #30210: +// the function takes the vocab and returns a common_chat_session), so +// opt.tmpls and the vocab can remain nullptr safely. // ============================================================ namespace { // Minimal helper: build body + options + out_files for early-throw tests std::vector g_out_files; +// The new signature with a throwaway session and no vocab -- every test below throws before +// either is touched. +json parse_chat_params(json &body, const server_chat_params &opt, std::vector &files) { + common_chat_session session; + return oaicompat_chat_params_parse(nullptr, body, opt, files, session); +} + json make_chat_body_with_messages(const json &messages_override = json::array({{{"role", "user"}, {"content", "hello"}}})) { return json{{"messages", messages_override}}; @@ -1021,14 +1029,14 @@ TEST(OaicompatChatParams, MissingMessages_Throws) { json body = {{"model", "x"}}; server_chat_params opt = make_no_jinja_opts(); std::vector files; - EXPECT_THROW(oaicompat_chat_params_parse(body, opt, files), std::exception); + EXPECT_THROW(parse_chat_params(body, opt, files), std::exception); } TEST(OaicompatChatParams, MessagesNotArray_Throws) { json body = {{"messages", "not-an-array"}}; server_chat_params opt = make_no_jinja_opts(); std::vector files; - EXPECT_THROW(oaicompat_chat_params_parse(body, opt, files), std::exception); + EXPECT_THROW(parse_chat_params(body, opt, files), std::exception); } TEST(OaicompatChatParams, NonAssistantMissingContent_Throws) { @@ -1036,7 +1044,7 @@ TEST(OaicompatChatParams, NonAssistantMissingContent_Throws) { json body = {{"messages", json::array({{{"role", "user"}}})}}; server_chat_params opt = make_no_jinja_opts(); std::vector files; - EXPECT_THROW(oaicompat_chat_params_parse(body, opt, files), std::exception); + EXPECT_THROW(parse_chat_params(body, opt, files), std::exception); } TEST(OaicompatChatParams, AssistantMissingBothContentAndToolCalls_Throws) { @@ -1044,7 +1052,7 @@ TEST(OaicompatChatParams, AssistantMissingBothContentAndToolCalls_Throws) { json body = {{"messages", json::array({{{"role", "assistant"}}})}}; server_chat_params opt = make_no_jinja_opts(); std::vector files; - EXPECT_THROW(oaicompat_chat_params_parse(body, opt, files), std::exception); + EXPECT_THROW(parse_chat_params(body, opt, files), std::exception); } TEST(OaicompatChatParams, ToolsWithoutJinja_Throws) { @@ -1052,14 +1060,14 @@ TEST(OaicompatChatParams, ToolsWithoutJinja_Throws) { {"tools", json::array({{{"type", "function"}}})}}; server_chat_params opt = make_no_jinja_opts(); std::vector files; - EXPECT_THROW(oaicompat_chat_params_parse(body, opt, files), std::exception); + EXPECT_THROW(parse_chat_params(body, opt, files), std::exception); } TEST(OaicompatChatParams, NonAutoToolChoiceWithoutJinja_Throws) { json body = {{"messages", json::array({{{"role", "user"}, {"content", "hi"}}})}, {"tool_choice", "none"}}; server_chat_params opt = make_no_jinja_opts(); std::vector files; - EXPECT_THROW(oaicompat_chat_params_parse(body, opt, files), std::exception); + EXPECT_THROW(parse_chat_params(body, opt, files), std::exception); } TEST(OaicompatChatParams, GrammarAndJsonSchema_Throws) { @@ -1068,7 +1076,7 @@ TEST(OaicompatChatParams, GrammarAndJsonSchema_Throws) { {"json_schema", {{"type", "object"}}}}; server_chat_params opt = make_no_jinja_opts(); std::vector files; - EXPECT_THROW(oaicompat_chat_params_parse(body, opt, files), std::exception); + EXPECT_THROW(parse_chat_params(body, opt, files), std::exception); } TEST(OaicompatChatParams, InvalidResponseFormatType_Throws) { @@ -1076,7 +1084,7 @@ TEST(OaicompatChatParams, InvalidResponseFormatType_Throws) { {"response_format", {{"type", "invalid_type"}}}}; server_chat_params opt = make_no_jinja_opts(); std::vector files; - EXPECT_THROW(oaicompat_chat_params_parse(body, opt, files), std::exception); + EXPECT_THROW(parse_chat_params(body, opt, files), std::exception); } TEST(OaicompatChatParams, ContentPartTypeUnsupported_Throws) { @@ -1084,7 +1092,7 @@ TEST(OaicompatChatParams, ContentPartTypeUnsupported_Throws) { {"content", json::array({{{"type", "video_url"}, {"url", "x"}}})}}})}}; server_chat_params opt = make_no_jinja_opts(); std::vector files; - EXPECT_THROW(oaicompat_chat_params_parse(body, opt, files), std::exception); + EXPECT_THROW(parse_chat_params(body, opt, files), std::exception); } TEST(OaicompatChatParams, ImageUrlWithoutAllowImage_Throws) { @@ -1096,7 +1104,7 @@ TEST(OaicompatChatParams, ImageUrlWithoutAllowImage_Throws) { server_chat_params opt = make_no_jinja_opts(); opt.allow_image = false; std::vector files; - EXPECT_THROW(oaicompat_chat_params_parse(body, opt, files), std::exception); + EXPECT_THROW(parse_chat_params(body, opt, files), std::exception); } TEST(OaicompatChatParams, ContentNotStringOrArray_Throws) { @@ -1104,7 +1112,7 @@ TEST(OaicompatChatParams, ContentNotStringOrArray_Throws) { json body = {{"messages", json::array({{{"role", "user"}, {"content", 42}}})}}; server_chat_params opt = make_no_jinja_opts(); std::vector files; - EXPECT_THROW(oaicompat_chat_params_parse(body, opt, files), std::exception); + EXPECT_THROW(parse_chat_params(body, opt, files), std::exception); } // ============================================================ From b440aac59e2cbf9d624470edf628567d8abbcea1 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 9 Oct 2026 19:21:45 +0000 Subject: [PATCH 3/8] Upgrade llama.cpp from b11531 to b11534 All eleven patches apply unchanged (verified in order against the pristine tag); see docs/history/llama-cpp-breaking-changes.md for the range. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01AytmJF9faEiQEVt6eetQS2 --- CLAUDE.md | 8 ++++---- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 ++ llama/CMakeLists.txt | 2 +- .../java/net/ladenthin/llama/value/LlamaCppVersion.java | 8 ++++---- 5 files changed, 12 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index b3261b7dd..8ee4f75a3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b11531** +Current llama.cpp pinned version: **b11534** ## Natives jars: one directory per backend (`.github/natives.csv`) @@ -897,7 +897,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network for the asset build; the embed step is plain cmake -P -git clone --depth 1 --branch b11531 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b11534 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build ) mkdir -p webui-generated /tmp/ui-gen cmake -DUI_SOURCE_DIR=/tmp/lc/tools/ui -DUI_BINARY_DIR=/tmp/ui-gen \ @@ -937,7 +937,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b11531`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b11534`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -2098,7 +2098,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b11531`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b11534`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.18.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index 9c888c90c..80e11a73d 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b11531](https://img.shields.io/badge/llama.cpp-%23b11531-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b11531) +[![llama.cpp b11534](https://img.shields.io/badge/llama.cpp-%23b11534-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b11534) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 38c732e98..8b67ec159 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -864,3 +864,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b11529–b11530 | patches + upstream verification | **All eleven patches apply unchanged**, verified in order against pristine b11530 with `git apply`. No patch-target file in the range (`0012` and `0016` touch `src/llama-model.cpp`, not the two files here), so every drop-check of the b11521–b11529 row holds unchanged. | | b11530–b11531 | One commit, 41 files, ~562/736 lines (119 KiB, forced single step: the one tag between b11530 and b11532). **#30210 refactors the chat API around `common_chat_session`** (`common/chat.{h,cpp}`): one object per generation that builds the prompt and owns the parser state (`feed()`/`finish()`), replacing the parser parameters that used to travel as request fields. Consequences on the server side, each of which reaches this project: `oaicompat_chat_params_parse()` takes the vocab and an `out_session` and no longer writes `chat_format`, `chat_parser`, `generation_prompt` or `parse_tool_calls` into the request JSON; `server_schema` drops those four fields (`echo` stays, parsed for validation only) -- none of them was a `RequestField` constant, so the Java wire surface is unchanged and `test_wire_contracts` keeps passing; `task_params::chat_parser_params` becomes `chat_format` + `reasoning_format` (`reasoning_in_content` in `to_json()` is now derived from `stream` and the legacy DeepSeek format, `generation_prompt` from `sampling.generation_prompt`); `server_task::create_state()` is gone and `server_response_reader::post_tasks()` takes the session, from which it builds every `task_result_state`; `server_task::apply_chat_session()` applies the session's sampling, stops and message spans. `common_chat_template` now picks its dedicated handler or runs the differential autoparser once per template (`params_init`/`analysis`), `common_chat_msg_delimiters::to_json()`/`_parse()` and `peg-parser`'s JSON serialisation are removed, and `common_sampling_add_preserved_tokens` / `common_sampling_add_grammar_triggers` move the tokenisation out of the schema. **Project source changes:** `jllama.cpp`'s `parse_oai_chat_params()` returns the session, the two dispatchers call `apply_chat_session()` and post through the session-taking `post_tasks()` (a default session for the non-chat paths, where it is a no-op); `test_utils.cpp`'s eleven early-validation tests call the new signature through one helper (the throws still fire before the session is built, so `tmpls` and the vocab stay null); `test_server.cpp`'s `task_params::to_json()` tests set `sampling.generation_prompt` and `stream` + `reasoning_format` instead of the removed members. Request-field diff (the mechanical check): `-chat_format -chat_parser -generation_prompt -parse_tool_calls`, `echo` from `field_bool` to `field_json`; response keys of `server-task.cpp` identical. | | b11530–b11531 | patches + upstream verification | **All eleven patches apply unchanged**, verified in order against pristine b11531 with `git apply` and by a fresh configure, build and `ctest` (603/603) through the real `FetchContent` path, since this step changes project code. Patch-target files in the range: `tools/server/server-context.cpp` and `server-context.h` (`0002`, `0003`: the session plumbing sits in `handle_completions_impl` and the route lambdas, away from `load_model` and the similarity getter/setter). Drop-checks unchanged. | +| b11531–b11534 | Three commits, 12 files, ~297/33 lines (32 KiB, the final step). **#30176 fixes OpenCL kernel compilation on Adreno A6x** (`ggml-opencl.cpp`, three kernels): the A6x shader compiler crashes on `kernel_cpy_f32_f32_pack`'s argument count and mis-folds `get_local_size` in the GEMV kernels, so the pack kernel is skipped and the GEMV kernels get a constant-fold workaround on that family -- this is the `opencl-android-aarch64` jar and the `llama-android-opencl` AAR on A6x devices. **#30108 uses exact GELU for ModernBERT encoders** (`src/models/modern-bert.cpp`, `llama-graph`, `llama-model.cpp`; tanh-GELU aliases kept on `ggml_geglu`): a numerics change for ModernBERT embedding and reranking models, not for the CI reranker (JinaBERT). **#29807 removes redundant CUDA copies after `SSM_SCAN`** (the state snapshot is written into the recurrent cache by the kernel; Mamba-style models on the CUDA jars). **No project source change.** | +| b11531–b11534 | patches + upstream verification | **All eleven patches apply unchanged**, verified in order against pristine b11534 with `git apply` and by a fresh configure through the real `FetchContent` path. Patch-target file in the range: `src/llama-model.cpp` (`0012`, `0016`: the GELU change is in the ModernBERT graph wiring, away from `load_tensors`' split block and the Kolibri-1 entries). Drop-checks against pristine b11534 -- the final state of the bump: `common_params_parse` still carries the `#ifdef _WIN32` `argv = utf8.ptrs.data()` override and `common_params_parse_main` is absent (`0001`); the load-progress assignment is unconditional (`0002`); no `slot_prompt_similarity` getter (`0003`); no embedded-mode, attach or worker-command counterpart (`0006`-`0008`); no `split_sum == 0` guard (`0012`); `common/log.h` names only `common_log_default_callback`, no sink (`0014`); no `stop_server` and the connect failure still aborts in `ggml-rpc.cpp` (`0015`); no `kolibri` under `src/` (`0016`); the four `_mm_prefetch` calls in `quants.c` are still uncast element arithmetic (`0017`). No `.github/` change upstream in the range. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index a7f34eaad..81f269bf8 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -261,7 +261,7 @@ endif() FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b11531 + GIT_TAG b11534 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index 0ebca1f49..b8f6c0255 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -9,13 +9,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b11531"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b11534"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b11531-"} — call + * plus the resolved upstream commit, e.g. {@code "b11534-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -23,14 +23,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b11531"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b11534"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b11531"; + public static final String LLAMA_CPP_VERSION = "b11534"; // Constants holder — not instantiable. private LlamaCppVersion() {} From e9758054a7f44e9714e70e1ea9d03a6b82529cc5 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 9 Oct 2026 19:29:20 +0000 Subject: [PATCH 4/8] Print the loader's two diagnostic lines to stderr A router worker JVM (NativeServer.setWorkerCommand) printed "[jllama] using native backend ..." onto the router's command pipe, which the router reports as unexpected output since llama.cpp b11401; the stdout of a java -jar server is otherwise upstream's. The CI smokes read both streams already. Closes the TODO entry. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01AytmJF9faEiQEVt6eetQS2 --- CHANGELOG.md | 5 +++++ TODO.md | 11 ----------- .../java/net/ladenthin/llama/loader/LlamaLoader.java | 4 ++-- 3 files changed, 7 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 69147ddc3..7a3fdecbf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -255,6 +255,11 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by 4.4.7 (none of the files the agent's carried JLine fixes touch changed between 4.4.6 and 4.4.7). ### Fixed +- **`LlamaLoader` prints its two diagnostic lines to stderr**, `[jllama] using native backend '…'` and + `[jllama] extracted '…'`, where they went to stdout before. A router worker JVM + (`NativeServer.setWorkerCommand`) printed them onto the router's command pipe, which since llama.cpp b11401 + the router reports as `unexpected output on the command pipe` (harmless, misleading); and the stdout of a + `java -jar` server is otherwise the upstream server's. The CI smokes read both streams already. - **`llama/patches/0017`** fixes two defects in the four `_mm_prefetch` calls of ggml's x86 `quants.c` (the SSSE3-without-AVX branch of `ggml_vec_dot_q4_0_q8_0`, and the only `_mm_prefetch` calls in the whole ggml tree), both by routing the address through one new diff --git a/TODO.md b/TODO.md index 83c553bbf..8433551fb 100644 --- a/TODO.md +++ b/TODO.md @@ -295,17 +295,6 @@ the server closed. Since b11361 `LlamaModel` holds a `server_routes` of its own `0007` and `native_server.cpp`. Needs a test with sleep enabled (`IdleSleepWakeIntegrationTest` is the template) before the fix, to show it red first. -### Router workers print the backend line onto the router's command pipe (cosmetic, since b11401) - -Since llama.cpp b11401 (#29895) a router child keeps its stdout for the state commands to the router -and redirects everything else written to stdout to stderr -- but only once `llama_server()` starts. -A JVM worker (`NativeServer.setWorkerCommand`, `patches/0008`) prints `LlamaLoader`'s -`[jllama] using native backend '...'` line to `System.out` before that, so the router logs it as -`unexpected output on the command pipe`. Harmless (the router warns and goes on), but misleading. -Moving the line to `System.err` would fix it; three smoke scripts grep for it -(`smoke-test-fatjar.sh` reads both streams, `smoke-rpc-fatjar.sh` and `smoke-natives-jars.sh` need -checking first), so it is not a one-line change. - ### LlamaLoader extraction-directory isolation (optional follow-up, low priority) Left over from the 2026-06-20 code audit (18/18 findings fixed in PRs #258/#260, regression tests in diff --git a/llama/src/main/java/net/ladenthin/llama/loader/LlamaLoader.java b/llama/src/main/java/net/ladenthin/llama/loader/LlamaLoader.java index 7b57173b4..c56ddb45e 100644 --- a/llama/src/main/java/net/ladenthin/llama/loader/LlamaLoader.java +++ b/llama/src/main/java/net/ladenthin/llama/loader/LlamaLoader.java @@ -279,7 +279,7 @@ private static void loadNativeLibrary(String name) { continue; } if (tryLoadBackend(backendResourcePath, backend, residentExtraFiles)) { - System.out.println("[jllama] using native backend '" + backend + "'"); + System.err.println("[jllama] using native backend '" + backend + "'"); return; } triedPaths.add(backendResourcePath); @@ -388,7 +388,7 @@ public static boolean loadNativeLibrary(Path path) { permissionSetter.apply(extractedFile); extractedFile.deleteOnExit(); - System.out.println("Extracted '" + fileName + "' to '" + extractedFilePath + "'"); + System.err.println("[jllama] extracted '" + fileName + "' to '" + extractedFilePath + "'"); return extractedFilePath; } catch (IOException e) { System.err.println(e.getMessage()); From a16494470659eae2e50e22b156a37fa2d5e9f698 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 9 Oct 2026 19:29:20 +0000 Subject: [PATCH 5/8] Require 1800 executed tests in every Java test job verify-test-counts.sh gains --min-executed (run minus skipped), the shape the run count cannot see: method-level assumptions skipping in bulk. Measured on a green run at b11529, the jobs execute 1856 (Windows) to 1865 (Linux) tests, a checkout without the models 1589. Both workflows pass --min-executed 1800 instead of --min-total 1500; closes the TODO entry. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01AytmJF9faEiQEVt6eetQS2 --- .github/verify-test-counts.sh | 53 +++++++++++++++++++++++--------- .github/workflows/java-tests.yml | 2 +- .github/workflows/publish.yml | 2 +- CHANGELOG.md | 4 +++ CLAUDE.md | 4 ++- TODO.md | 11 ++----- 6 files changed, 49 insertions(+), 27 deletions(-) diff --git a/.github/verify-test-counts.sh b/.github/verify-test-counts.sh index 7bb818148..bdbc7b991 100755 --- a/.github/verify-test-counts.sh +++ b/.github/verify-test-counts.sh @@ -14,33 +14,39 @@ # The shape is what defeats the obvious guard: a CLASS-LEVEL assumption failure makes Surefire # record tests="0" errors="0" failures="0" skipped="0" for that class. It contributes NO test # entries at all, so "did this run skip anything?" is structurally blind to it — a skipped test is -# still a reported test. Two checks catch it: +# still a reported test. Three checks catch it: # # 1. Any testsuite reporting tests="0". This is the exact signature above, it is precise, and it # is platform-independent: a class that runs nowhere is a bug on every OS. -# 2. A floor on the total number of tests executed. The backstop for a whole class file going -# missing from the run rather than reporting zero. +# 2. A floor on the number of tests EXECUTED (run minus skipped). This is the other shape of the +# same failure: a METHOD-LEVEL assumption (Assume.assumeTrue inside a test) does report the +# test, as skipped. Measured on a green run at b11529, every test-java-* job runs 1869 tests +# and skips 4 (Linux), 11 (macOS, all three jobs) or 13 (Windows, Ninja and MSVC), i.e. it +# executes 1856 to 1865; the same tree in a checkout WITHOUT the models runs 1869 and skips +# 280, i.e. executes 1589. The jobs pass --min-executed 1800: that catches the model set going +# missing (or a resolver regression) with a margin of ~200 either way, and tolerates ordinary +# test removals of up to ~56 tests before someone has to lower it, deliberately. +# 3. A floor on the total (--min-total), the backstop for a whole class file going missing from +# the run rather than reporting zero. Implied by check 2 when both are given. # -# Check 1 is the sensitive one; check 2 is deliberately slack so it never fails spuriously on a -# platform that legitimately runs fewer tests. Tighten --min-total once real per-platform numbers -# are known from a green run. -# -# Usage: .github/verify-test-counts.sh [--min-total N] +# Usage: .github/verify-test-counts.sh [--min-total N] [--min-executed N] # Exit codes: 0 all good, 1 a check failed, 2 nothing to scan. set -euo pipefail REPORT_DIR="${1:-}" MIN_TOTAL=0 +MIN_EXECUTED=0 shift || true while [ $# -gt 0 ]; do case "$1" in --min-total) MIN_TOTAL="$2"; shift 2 ;; + --min-executed) MIN_EXECUTED="$2"; shift 2 ;; *) echo "unknown argument: $1" >&2; exit 1 ;; esac done -[ -n "$REPORT_DIR" ] || { echo "usage: $0 [--min-total N]" >&2; exit 1; } +[ -n "$REPORT_DIR" ] || { echo "usage: $0 [--min-total N] [--min-executed N]" >&2; exit 1; } [ -d "$REPORT_DIR" ] || { echo "ERROR: no such directory: $REPORT_DIR" >&2; exit 2; } shopt -s nullglob @@ -54,18 +60,26 @@ if [ "${#reports[@]}" -eq 0 ]; then exit 2 fi +# An attribute of the element; the first match, so a nested element carrying the same +# attribute name cannot shift the number ( has neither, a skipped test is a child element). +suite_attr() { + local n + n="$(grep -o "$2=\"[0-9]*\"" "$1" | head -1 | grep -o '[0-9]*' || true)" + echo "${n:-0}" +} + total=0 +skipped=0 empty_suites=() for f in "${reports[@]}"; do - # The count lives on the element; take the first match so a nested element - # carrying the same attribute name cannot shift the number. - n="$(grep -o 'tests="[0-9]*"' "$f" | head -1 | grep -o '[0-9]*' || true)" - [ -n "$n" ] || n=0 + n="$(suite_attr "$f" tests)" total=$((total + n)) + skipped=$((skipped + $(suite_attr "$f" skipped))) if [ "$n" -eq 0 ]; then empty_suites+=("$(basename "$f")") fi done +executed=$((total - skipped)) status=0 @@ -79,11 +93,20 @@ if [ "${#empty_suites[@]}" -gt 0 ]; then fi if [ "$total" -lt "$MIN_TOTAL" ]; then - echo "ERROR: only $total test(s) executed, below the floor of $MIN_TOTAL" >&2 + echo "ERROR: only $total test(s) run, below the floor of $MIN_TOTAL" >&2 + status=1 +fi + +if [ "$executed" -lt "$MIN_EXECUTED" ]; then + echo "ERROR: only $executed test(s) executed ($total run, $skipped skipped), below the floor of $MIN_EXECUTED" >&2 + echo " Method-level assumptions skipping in bulk look like this: a model file the tests expect is" >&2 + echo " missing or resolves to the wrong directory (TestConstants.resolveModelPath), or a native" >&2 + echo " library failed to load. If the suite legitimately shrank, lower --min-executed in the" >&2 + echo " workflow deliberately." >&2 status=1 fi if [ "$status" -eq 0 ]; then - echo "test counts verified: $total test(s) across ${#reports[@]} class(es), none empty (floor $MIN_TOTAL)" + echo "test counts verified: $total test(s) run, $skipped skipped, $executed executed across ${#reports[@]} class(es), none empty (floors: total $MIN_TOTAL, executed $MIN_EXECUTED)" fi exit "$status" diff --git a/.github/workflows/java-tests.yml b/.github/workflows/java-tests.yml index c3fd7ed79..94d0e106d 100644 --- a/.github/workflows/java-tests.yml +++ b/.github/workflows/java-tests.yml @@ -93,7 +93,7 @@ jobs: # zero check is the sensitive half. - name: Verify tests actually ran shell: bash - run: .github/verify-test-counts.sh llama/target/surefire-reports --min-total 1500 + run: .github/verify-test-counts.sh llama/target/surefire-reports --min-executed 1800 - name: Memory after tests if: always() && runner.os == 'macOS' run: vm_stat && sysctl hw.memsize hw.physmem diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 53c49d6cc..9b9676679 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -2329,7 +2329,7 @@ jobs: # muted on every test-java-* job for months. The floor is deliberately slack; the per-class # zero check is the sensitive half. - name: Verify tests actually ran - run: .github/verify-test-counts.sh llama/target/surefire-reports --min-total 1500 + run: .github/verify-test-counts.sh llama/target/surefire-reports --min-executed 1800 - uses: actions/upload-artifact@v7 if: success() with: diff --git a/CHANGELOG.md b/CHANGELOG.md index 7a3fdecbf..2903fa9f8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -57,6 +57,10 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by in `build.sh` opts back in). With CUDA 13.4 and sccache 0.18.0 the nvcc launcher failed on the first `.cu` files in two confirmed runs (`Missing "cubin" file output`, `Compiler killed by signal 126`), and the uncached retry that followed made those jobs slower than a cold build; the C/C++ objects stay cached. +- **CI: every `test-java-*` job now requires at least 1800 tests *executed*** (`verify-test-counts.sh + --min-executed`, run minus skipped), where it required 1500 *run* before. The run count cannot see the + other shape of the muted-suite failure -- method-level assumptions skipping in bulk -- while the executed + count does: the jobs execute 1856 (Windows) to 1865 (Linux) tests, a checkout without the models 1589. - **Upgraded the pinned llama.cpp from b11512 to b11529**, in 5 reviewed steps, each ending at a tag. No carried patch needed a refresh, and every drop-check still finds its defect. In the range: the server's default port moves from 8080 to 9931 (#30159, the `NativeServer` entry above); CUDA top-k selection is diff --git a/CLAUDE.md b/CLAUDE.md index 8ee4f75a3..35e3fdb93 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1996,7 +1996,9 @@ as *nothing at all* while the job still went green, on every `test-java-*` job. shape, because it defeats the obvious guard: a class-level `@BeforeAll` assumption makes Surefire record `tests="0" errors="0" skipped="0"` — the class contributes **no** test entries, so a check of the form "did this run skip anything?" is blind to it. The only thing that catches it directly is a -floor on the number of tests actually executed (see `TODO.md`). It is why several stale +floor on the number of tests actually executed -- `.github/verify-test-counts.sh`, run by every +`test-java-*` job: no class with zero entries, and at least 1800 tests executed (run minus skipped; +the jobs execute 1856 to 1865, a checkout without models 1589). It is why several stale assertions (e.g. `LlamaModelTest#testGetMetrics` against a payload shape upstream had dropped at b10408) never failed in CI. The fix is **`TestConstants.resolveModelPath` / `resolveModelProperty`**, which accept either layout — module-relative first, then the reactor root diff --git a/TODO.md b/TODO.md index 8433551fb..a2f4b6b07 100644 --- a/TODO.md +++ b/TODO.md @@ -590,8 +590,8 @@ and have only run locally so far. > **Update.** The `IdleSleepWakeIntegrationTest` added to close the `wake_and_post` gap immediately > found a real JVM crash (SIGSEGV on all six CI platforms) — see the CHANGELOG "Fixed" entry. Both > facets are fixed in that PR via the `wake_server()` choke point. This is the clearest evidence for -> the entry below about a floor on executed tests: the defect had been reachable from public API for -> as long as `--sleep-idle-seconds` has existed, and nothing ran that path. +> the floor on executed tests that `.github/verify-test-counts.sh` enforces since: the defect had been +> reachable from public API for as long as `--sleep-idle-seconds` has existed, and nothing ran that path. A mutation pass over the branch applied 27 mutations and 26 went red on the test that claims them, so no test here passes with its subject deleted. What it did find is code with **no runnable guard**. @@ -610,13 +610,6 @@ away with the patch itself at the b11080 bump. This is what remains. `resolveModelPath(literal)` against a `@TempDir` fixture planted at the reactor root, so the assertion does not depend on a real model being present. -- **Nothing asserts a floor on the number of tests actually executed.** A class-level `@BeforeAll` - assumption makes Surefire record `tests="0" errors="0" skipped="0"` — the class contributes no - entries at all, so "did the run skip anything?" is structurally blind to it. This is exactly how - the model-gated suite stayed silently muted for months. Summing `tests=` across - `target/surefire-reports/TEST-*.xml` in each `test-java-*` job and failing below a pinned minimum - is the one check that would have caught it directly, and it is cheap. - ### Test-coverage debt found during the b10649 review (PR #403) Each item below was verified against pristine upstream tags and is real, but none is a regression From 1a71b4b939e7dd0ecf148de251494a0597f14935 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 9 Oct 2026 19:29:20 +0000 Subject: [PATCH 6/8] Add a one-file JBang chat example examples/jbang/Chat.java runs with `jbang model.gguf` and no project. Its //DEPS lines name the classes jar and the CPU natives jar of every desktop platform themselves: JBang treats a pom dependency such as llama-platform as a BOM and puts nothing of it on the classpath (measured). check-natives.py holds the lines to the platform=yes rows of natives.csv and the version to the README's install snippet, with a unit test. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01AytmJF9faEiQEVt6eetQS2 --- .github/buildcheck/natives.py | 32 ++++++++++ .github/buildcheck/tests/test_natives.py | 21 +++++++ CHANGELOG.md | 6 ++ CLAUDE.md | 2 + README.md | 15 +++++ examples/jbang/Chat.java | 76 ++++++++++++++++++++++++ 6 files changed, 152 insertions(+) create mode 100644 examples/jbang/Chat.java diff --git a/.github/buildcheck/natives.py b/.github/buildcheck/natives.py index d127c79a5..b35ae9da2 100644 --- a/.github/buildcheck/natives.py +++ b/.github/buildcheck/natives.py @@ -247,6 +247,37 @@ def check_readme(natives, readme_text): return failures + compare("README.md all-backends fat jars", fatjar_targets(natives), fatjar_names(readme_text)) +JBANG_EXAMPLE = "examples/jbang/Chat.java" + + +def check_jbang_example(natives, script_text, readme_text): + """The one-file JBang example runs without a checkout, so it cannot take `llama-platform`: JBang + treats a `pom` dependency as a BOM and puts nothing of it on the classpath (measured). It therefore + names the classes jar and the platform=yes natives jars itself -- exactly those, at the version of + the README's install snippet, which the version bump moves by hand.""" + deps = [line.split()[1] for line in script_text.splitlines() if line.startswith("//DEPS ")] + coords = [d.split(":") for d in deps] + good = [c for c in coords if c[:2] == ["net.ladenthin", "llama"] and len(c) in (3, 4)] + failures = [f"{JBANG_EXAMPLE}: `{d}` is not net.ladenthin:llama:[:]" + for d, c in zip(deps, coords) if c not in good] + match = re.search(r"llama-platform\s*([^<]+)", readme_text) + expected = match.group(1) if match else None + versions = {c[2] for c in good} + if versions != {expected}: + failures.append(f"{JBANG_EXAMPLE}: DEPS version(s) {sorted(versions)} must be the README install " + f"snippet's {expected}") + if not any(len(c) == 3 for c in good): + failures.append(f"{JBANG_EXAMPLE}: the classes jar net.ladenthin:llama: is missing") + named = [c[3] for c in good if len(c) == 4] + platform = [r["classifier"] for r in natives if r["platform"] == "yes"] + failures += [f"{JBANG_EXAMPLE}: missing the platform natives jar {c}" for c in platform if c not in named] + failures += [f"{JBANG_EXAMPLE}: names {c}, which is not a platform=yes row of .github/natives.csv" + for c in named if c not in platform] + if len(set(named)) != len(named): + failures.append(f"{JBANG_EXAMPLE}: a natives jar is named twice") + return failures + + def check_agent_class_path(natives, agent_pom_text): """`java -jar` on the agent jar finds the core through its manifest Class-Path, which names every all-backends fat jar.""" @@ -284,5 +315,6 @@ def check(root): + check_cmake(natives, read(root, "llama/CMakeLists.txt")) + check_dependency_allowlist(natives, nativedeps.ALLOWED) + check_readme(natives, read(root, "README.md")) + + check_jbang_example(natives, read(root, JBANG_EXAMPLE), read(root, "README.md")) + check_agent_class_path(natives, read(root, "llama-atmosphere-agent/pom.xml")) + check_agent_version(read(root, "pom.xml"), read(root, "llama-atmosphere-agent/pom.xml"))) diff --git a/.github/buildcheck/tests/test_natives.py b/.github/buildcheck/tests/test_natives.py index d1194015d..ef64eb015 100644 --- a/.github/buildcheck/tests/test_natives.py +++ b/.github/buildcheck/tests/test_natives.py @@ -193,6 +193,27 @@ def test_readme(self): failures = natives.check_readme(ROWS, text.replace("`cpu-linux-s390x`", "cpu-linux-s390x")) self.assertEqual(failures, ["README.md does not document the natives jar `cpu-linux-s390x`"]) + def test_jbang_example(self): + readme = "llama-platform\n 5.2.0\n" + platform = [r["classifier"] for r in ROWS if r["platform"] == "yes"] + lines = ["//DEPS net.ladenthin:llama:5.2.0"] + [f"//DEPS net.ladenthin:llama:5.2.0:{c}" for c in platform] + script = "///usr/bin/env jbang\n" + "\n".join(lines) + "\n// a comment naming net.ladenthin:llama:5.2.0:cuda13-linux-x86-64 is not a dependency\n" + self.assertEqual(natives.check_jbang_example(ROWS, script, readme), []) + # a platform jar missing, a GPU jar named instead + broken = script.replace(":cpu-linux-s390x", ":cuda13-linux-x86-64") + self.assertEqual(natives.check_jbang_example(ROWS, broken, readme), + ["examples/jbang/Chat.java: missing the platform natives jar cpu-linux-s390x", + "examples/jbang/Chat.java: names cuda13-linux-x86-64, which is not a platform=yes row of .github/natives.csv"]) + # the classes jar missing + failures = natives.check_jbang_example(ROWS, script.replace("//DEPS net.ladenthin:llama:5.2.0\n", ""), readme) + self.assertEqual(failures, ["examples/jbang/Chat.java: the classes jar net.ladenthin:llama: is missing"]) + # a version the README has moved past (the bump edits both by hand) + failures = natives.check_jbang_example(ROWS, script, readme.replace("5.2.0", "5.3.0")) + self.assertEqual(failures, ["examples/jbang/Chat.java: DEPS version(s) ['5.2.0'] must be the README install snippet's 5.3.0"]) + # another artifact, or JBang's BOM-only pom form + failures = natives.check_jbang_example(ROWS, script + "//DEPS net.ladenthin:llama-platform:5.2.0@pom\n", readme) + self.assertEqual(failures, ["examples/jbang/Chat.java: `net.ladenthin:llama-platform:5.2.0@pom` is not net.ladenthin:llama:[:]"]) + def test_agent_class_path(self): good = "llama-${v}-all-linux-x86-64-jar-with-dependencies.jar llama-${v}-jar-with-dependencies.jar" self.assertEqual(natives.check_agent_class_path(ROWS, good), []) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2903fa9f8..45e286c28 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,12 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by ## [Unreleased] ### Added +- **`examples/jbang/Chat.java`**, a one-file console chat that [JBang](https://www.jbang.dev) runs without a + checkout or a build (`jbang https://github.com/bernardladenthin/java-llama.cpp/blob/main/examples/jbang/Chat.java + model.gguf`). Its `//DEPS` lines name the classes jar and the CPU natives jar of every desktop platform + themselves -- JBang treats a `pom` dependency such as `llama-platform` as a BOM and puts nothing of it on the + classpath -- and `check-natives.py` holds them to the `platform=yes` rows of `natives.csv` and to the README's + release version. - **`ModelParameters.setMoeCacheMib(int)`** (`--moe-cache-mib`, llama.cpp b11480, EXPERIMENTAL upstream): a GPU cache for the MoE expert weights a `setCpuMoeLayers` / `--cpu-moe` setup keeps in host memory, split among several GPUs like the layers; `0` (the default) disables it. diff --git a/CLAUDE.md b/CLAUDE.md index 35e3fdb93..d6df1dd3a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2492,6 +2492,8 @@ missed again.) (The per-classifier snippets were **deduplicated** to a single canonical + template pair, so the release version now appears in only ~4 spots here, not ~20 — the runtime details live once in the classifier table.) +- **`examples/jbang/Chat.java`** — the version in its `//DEPS` lines; `check-natives.py` holds it to + the README install snippet's version, so forgetting it fails `code-style`. - **`llama-langchain4j/README.md`** — its own `` snippet. - **`llama-atmosphere-agent/pom.xml`** — its own ``, which must equal the reactor's (standalone project outside the reactor, so `versions:set` skips it; `check-natives.py` fails until diff --git a/README.md b/README.md index 80e11a73d..e4b554593 100644 --- a/README.md +++ b/README.md @@ -149,6 +149,21 @@ In Gradle it is just `implementation("net.ladenthin:llama-platform:5.2.0")`.) There are multiple [examples](llama/src/test/java/examples). +### Try it with JBang, no project needed + +[`examples/jbang/Chat.java`](examples/jbang/Chat.java) is a one-file console chat that +[JBang](https://www.jbang.dev) runs straight from the repository, with any JDK 8+ and a GGUF of an +instruction-tuned model: + +```bash +jbang https://github.com/bernardladenthin/java-llama.cpp/blob/main/examples/jbang/Chat.java model.gguf +``` + +Its `//DEPS` lines name the classes jar and the CPU natives jar of every desktop platform (the jars +`llama-platform` names; JBang treats a `pom` dependency as a BOM and puts nothing of it on the +classpath), and the loader picks this machine's. Copy the file as a starting point; a GPU backend is +one more `//DEPS` line (see [Choosing the natives jars](#choosing-the-natives-jars)). + ### Snapshot builds Every push to `main` publishes a snapshot to the [Sonatype Central snapshot repository](https://central.sonatype.com/repository/maven-snapshots/net/ladenthin/llama/). diff --git a/examples/jbang/Chat.java b/examples/jbang/Chat.java new file mode 100644 index 000000000..ab693d78f --- /dev/null +++ b/examples/jbang/Chat.java @@ -0,0 +1,76 @@ +///usr/bin/env jbang "$0" "$@" ; exit $? +// SPDX-FileCopyrightText: 2026 Bernard Ladenthin +// +// SPDX-License-Identifier: MIT +//JAVA 8+ +//DEPS net.ladenthin:llama:5.2.0 +//DEPS net.ladenthin:llama:5.2.0:cpu-linux-x86-64 +//DEPS net.ladenthin:llama:5.2.0:cpu-linux-aarch64 +//DEPS net.ladenthin:llama:5.2.0:cpu-linux-s390x +//DEPS net.ladenthin:llama:5.2.0:cpu-windows-x86-64 +//DEPS net.ladenthin:llama:5.2.0:cpu-windows-x86 +//DEPS net.ladenthin:llama:5.2.0:cpu-windows-aarch64 +//DEPS net.ladenthin:llama:5.2.0:metal-macos-aarch64 + +import java.io.BufferedReader; +import java.io.IOException; +import java.io.InputStreamReader; +import java.nio.charset.StandardCharsets; +import net.ladenthin.llama.LlamaModel; +import net.ladenthin.llama.Session; +import net.ladenthin.llama.parameters.ModelParameters; +import net.ladenthin.llama.value.LlamaOutput; + +/** + * A multi-turn chat on the console that JBang runs with no project and no build: + * + * 
+ *   jbang Chat.java path/to/an-instruct-model.gguf
+ *
+ * + * The {@code //DEPS} lines above are the classes jar and the CPU natives jars of every desktop platform + * -- the jars {@code net.ladenthin:llama-platform} names, listed here one by one because JBang treats a + * {@code pom} dependency as a BOM and puts nothing of it on the classpath. The loader picks the jar of + * this machine; the others are downloaded once and ignored. A GPU backend (for example + * {@code net.ladenthin:llama:5.2.0:cuda13-linux-x86-64}) is one more DEPS line and is tried before the CPU. + * + *

A {@link Session} keeps the conversation, so every turn passes only the new message, and the model's + * chat template is applied for it; each reply is streamed as it is generated. An empty line or the end + * of the input (Ctrl+D, on Windows Ctrl+Z) ends the chat. + */ +public class Chat { + + public static void main(String... args) throws IOException { + if (args.length != 1) { + System.err.println("usage: jbang Chat.java "); + System.exit(2); + } + BufferedReader console = new BufferedReader(new InputStreamReader(System.in, StandardCharsets.UTF_8)); + try (LlamaModel model = new LlamaModel(new ModelParameters().setModel(args[0]).setCtxSize(4096)); + // Slot 0 holds this conversation; the customizer applies to every turn. + Session session = new Session(model, 0, "You are a concise, helpful assistant.", + p -> p.withNPredict(512).withTemperature(0.7f))) { + while (true) { + System.out.print("\nYou: "); + String message = console.readLine(); + if (message == null || message.trim().isEmpty()) { + return; + } + System.out.print("Assistant: "); + StringBuilder reply = new StringBuilder(); + try { + for (LlamaOutput output : session.stream(message)) { + System.out.print(output.text); + reply.append(output.text); + } + } catch (RuntimeException e) { + // Without a committed reply the session would refuse every further turn. + session.cancelStream(); + throw e; + } + session.commitStreamedReply(reply.toString()); + System.out.println(); + } + } + } +} From bbc448a506f49803f1709f15a9f2e43a16aee2d7 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 9 Oct 2026 19:29:20 +0000 Subject: [PATCH 7/8] Record the b11529 to b11534 range in the CHANGELOG and the chat.h review row Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01AytmJF9faEiQEVt6eetQS2 --- CHANGELOG.md | 10 ++++++++++ CLAUDE.md | 2 +- 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 45e286c28..22a53ce91 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -63,6 +63,16 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by in `build.sh` opts back in). With CUDA 13.4 and sccache 0.18.0 the nvcc launcher failed on the first `.cu` files in two confirmed runs (`Missing "cubin" file output`, `Compiler killed by signal 126`), and the uncached retry that followed made those jobs slower than a cold build; the C/C++ objects stay cached. +- **Upgraded the pinned llama.cpp from b11529 to b11534**, in 3 reviewed steps, each ending at a tag. No + carried patch needed a refresh, and every drop-check still finds its defect. The one step with project + code is b11531, **#30210 "chat : refactor API"**: the prompt and the parser state of a chat generation now + live in a `common_chat_session` that `oaicompat_chat_params_parse` fills and the server task applies, + replacing the `chat_format` / `chat_parser` / `generation_prompt` / `parse_tool_calls` request fields + (none was a `RequestField`, so the Java wire surface is unchanged); `jllama.cpp` threads a session through + its four chat entry points and the C++ tests follow the new signatures. Also in the range: OpenCL kernels + that compile on Adreno A6x (#30176, the `opencl-android-aarch64` jar and the `llama-android-opencl` AAR on + that family), exact GELU for ModernBERT encoders (#30108), and no redundant CUDA copies after `SSM_SCAN` + (#29807). Per-step record: `docs/history/llama-cpp-breaking-changes.md`. - **CI: every `test-java-*` job now requires at least 1800 tests *executed*** (`verify-test-counts.sh --min-executed`, run minus skipped), where it required 1500 *run* before. The run count cannot see the other shape of the muted-suite failure -- method-level assumptions skipping in bulk -- while the executed diff --git a/CLAUDE.md b/CLAUDE.md index d6df1dd3a..31a9477c0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1440,7 +1440,7 @@ git show :tools/server/server-task.cpp | { grep -oE '\{ *"[A-Za-z_0-9.]+" * | File | What to watch for | |------|-------------------| | `common/common.h` | `common_params`/`common_params_speculative` struct fields, `model_alias` container type, `common_init_result` shape, `build_info` symbol (removed in b8831 — now `llama_build_info()` from `build-info.h`) | -| `common/chat.h` | `common_chat_parser_params` (was `common_chat_syntax`), `to_json_oaicompat`, `common_chat_msg_diff_to_json_oaicompat`, `set_tool_call_ids` | +| `common/chat.h` | `common_chat_session` (b11531, #30210: one object per generation owning the prompt and the parser state; `oaicompat_chat_params_parse` fills it as its last argument, `server_task::apply_chat_session` consumes it, `server_response_reader::post_tasks` takes it -- `jllama.cpp` threads one through every chat entry point), `common_chat_parser_params` (was `common_chat_syntax`), `to_json_oaicompat`, `common_chat_msg_diff_to_json_oaicompat`, `set_tool_call_ids` | | `common/speculative.h` | `common_speculative_init`, `common_speculative_draft`, `common_speculative_accept` signatures, struct names | | `tools/mtmd/mtmd.h` | `mtmd_context_params` fields, `image_marker`/`media_marker` API, deprecated symbols (was `common/mtmd.h` before ~b8190) | | `include/llama-cpp.h` | `common_init_result_ptr` type, access pattern changes (`.get()` vs `->method()`) | From 5865f4d2d4752b44bbe9c71b471977096ebda874 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 9 Oct 2026 19:29:20 +0000 Subject: [PATCH 8/8] Add the handover prompt for the local agent (PR A) What the branch contains, what the sandbox could not verify (models, Windows, GPU), the deliverables in order, and the branch protocol. Deleted again once PR A is merged. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01AytmJF9faEiQEVt6eetQS2 --- REUSE.toml | 1 + docs/handover/local-agent-pr-a.md | 95 +++++++++++++++++++++++++++++++ 2 files changed, 96 insertions(+) create mode 100644 docs/handover/local-agent-pr-a.md diff --git a/REUSE.toml b/REUSE.toml index 0622cfc41..da877bacb 100644 --- a/REUSE.toml +++ b/REUSE.toml @@ -13,6 +13,7 @@ path = [ "android-llmservice/CLAUDE.md", "CLAUDE_TEST_GENERATION_PROMPT.md", "docs/RELEASE.md", + "docs/handover/**", "docs/feature-investigation-llama-stack-client-kotlin.md", "docs/history/49be664_24918e4.md", "docs/history/49be664_open_issues.md", diff --git a/docs/handover/local-agent-pr-a.md b/docs/handover/local-agent-pr-a.md new file mode 100644 index 000000000..4a905cbd5 --- /dev/null +++ b/docs/handover/local-agent-pr-a.md @@ -0,0 +1,95 @@ +# Handover prompt for the local agent -- PR A on `claude/hopeful-pascal-9jlbqb` + +Paste the text below the line into the local agent (Windows workstation: Ryzen 7 5800H, RTX 3070, +AMD iGPU, MSVC + LLVM + Docker + JDK 21 + Maven). It describes what this branch already contains, +what the remote session cannot verify from its sandbox, and the exact deliverables. The file is +deleted again once PR A is merged. + +--- + +You are working in a checkout of `bernardladenthin/java-llama.cpp` on the branch +`claude/hopeful-pascal-9jlbqb`. Read `CLAUDE.md` first (it is long; the sections "Natives jars", +"CPU variants", "Building the native library for local Java tests" and "Testing" matter here). + +## What the branch contains (do not redo it) + +| Item | State | Where | +|---|---|---| +| **A1** llama.cpp b11529 → b11534 | done, 3 commits (b11530, b11531, b11534); 603/603 C++ tests and the model-free Java suite are green on Linux x86-64 | `llama/CMakeLists.txt`, `docs/history/llama-cpp-breaking-changes.md` | +| **A2** floor on *executed* tests | done: `.github/verify-test-counts.sh --min-executed 1800` in every `test-java-*` job | `.github/verify-test-counts.sh`, `java-tests.yml`, `publish.yml` | +| **A3** loader lines to stderr | done: `[jllama] using native backend '…'` and `[jllama] extracted …` print to `System.err` | `LlamaLoader.java` | +| **A4** one-file JBang chat | done: `examples/jbang/Chat.java` names the classes jar + the 7 desktop CPU natives jars (JBang treats a `pom` dependency as a BOM, so `llama-platform` cannot be used there); `check-natives.py` guards the lines | `examples/jbang/Chat.java`, README "Try it with JBang" | + +The sandbox the branch was written in has no models (HuggingFace is blocked), no Windows and no GPU. +Everything below is what only your machine can show. + +## Deliverables, in this order + +1. **Toolchain inventory** (once): clang versions and paths (LLVM on `PATH`, the one inside Visual + Studio under `VC\Tools\Llvm\x64`), Visual Studio and MSVC toolset versions, Windows SDK, Ninja, + sccache, JDK(s), Docker images you have pulled, CUDA toolkit and driver version, Vulkan SDK, + Android SDK/NDK. Two questions: do you have access to a Mac (package B1 needs one, otherwise it is + CI-only)? Do you have an Android device, ideally with a Qualcomm Adreno GPU (package E2)? + +2. **A1 -- the Java suite with models on Windows.** Build the Ninja CPU library at b11534 the way CI + does (`mvn -q compile` in `llama/`, then `.github\build.bat -G "Ninja Multi-Config" -DOS_NAME=Windows + -DOS_ARCH=x86_64 -DBUILD_TESTING=ON`, `ctest --test-dir build -C Release`), download the CI model set + (`.github/models.csv`, into `models/` at the repo root) and run `mvn -f llama/pom.xml test`. + Report: ctest result, Surefire's final `Tests run: N, Failures, Errors, Skipped` line, and the + names of the skipped tests (`grep -l '.dll`, + `ggml-rpc.dll`) and the content of `jllama-files.txt`; whether `jllama-extras.txt` exists. + - `python .github/verify-native-deps.py` over that directory: paste its output. + - The Java suite (`mvn -f llama/pom.xml test`) against the variant build, with models. + - Which variant ggml picks on the 5800H (Zen 3 has AVX2 and no AVX-512, so expect `haswell`; the + loader log at `-lv 4` names it), and `llama-bench`-style numbers through the library if cheap: pp512 and + tg128 for Qwen3-0.6B Q4_K_M with 8 threads, variant build versus the single static CPU build. + +## How to work on the branch + +- Pull before every push (`git pull --rebase=false origin claude/hopeful-pascal-9jlbqb`); never force-push, + never rewrite the remote session's commits. Commit only files you own: your reports go to + `docs/handover/local-agent-report-.md` (plain text, numbers in tables, command lines as + code blocks); do not edit source files on this branch unless a step above says so -- propose the + change in the report instead. End your commits with your own `Co-Authored-By` / session trailers, + not the remote session's. +- Before pushing anything that touches `.java`, `.cpp`/`.hpp` or `.github/`: `mvn -f llama/pom.xml + spotless:apply`, `clang-format` 23.1.3 on the C++ files, and the Python checks + (`python .github/check-natives.py`, `check-patches.py`, `check-release-gate.py`, + `check-run-scripts.py`, `check-shared-files.py`, `python -m unittest discover -s + .github/buildcheck/tests -t .github`). +- Do not dispatch or re-run CI workflows; the owner does that. +- Measurements are only useful as text: paste the console lines, not a description of them.