From 18f58b4aeef375029eac770bfb1b463da0a75e89 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 14 Oct 2026 13:19:37 +0000 Subject: [PATCH 1/3] Upgrade llama.cpp from b11612 to b11617 Five upstream commits: four Vulkan (bf16 mulmat activations, Intel Arc shader count, the Intel Windows free-memory workaround, a compile fix) and #30480, which hands the k-pool selection of glm5-next / qwen4exp to the attention in the mask layout. No patch-target file in the range; all thirteen patches apply unchanged, ctest 608/608, the model-free native Java tests 27/27 over the b11617 library. No project source change. The 0012 drop rule now keys on the fixing commit (df18bee11 / #29677) rather than on "past b11612": b11617 is past b11612 and still lacks the fix, and no tag up to b11618 carries it. Patch header, test header, CLAUDE.md row and the TODO.md section say so. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_017kVXQfPe1ACLAuk29Y4d2H --- CLAUDE.md | 14 +++++++------- README.md | 2 +- TODO.md | 6 +++--- docs/history/llama-cpp-breaking-changes.md | 4 +++- llama/CMakeLists.txt | 2 +- ...-zero-split-sum-and-name-the-device-index.patch | 8 ++++---- .../net/ladenthin/llama/value/LlamaCppVersion.java | 8 ++++---- llama/src/test/cpp/test_model_split.cpp | 9 +++++---- 8 files changed, 28 insertions(+), 25 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 607f216e..02c68801 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b11612** +Current llama.cpp pinned version: **b11617** ## Natives jars: library jars and GPU module jars (`.github/natives.csv`) @@ -178,7 +178,7 @@ measurements behind the Windows step are in `docs/handover/local-agent-report-b1 `add_subdirectory(ggml)` when a `ggml` target exists; a module-only build never adds llama.cpp). llama, mtmd and llama-common stay static in `libjllama.so`. **Every ggml default llama.cpp's own CMakeLists changes before its `add_subdirectory(ggml)` is repeated** -- `GGML_LLAMAFILE_DEFAULT` - and `GGML_CUDA_GRAPHS_DEFAULT` at b11612 -- because adding ggml first skips that block; re-check with + and `GGML_CUDA_GRAPHS_DEFAULT` at b11617 -- because adding ggml first skips that block; re-check with `grep -n _DEFAULT CMakeLists.txt` at the new tag on every bump. The second one was missing once: ggml's own default for CUDA graphs is OFF, and a CUDA module built without it lost two thirds of its token generation (265.6 -> 88.8 tg128 on an RTX 3070, prompt processing -20%; Vulkan, which @@ -1106,7 +1106,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network for the asset build; the embed step is plain cmake -P -git clone --depth 1 --branch b11612 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b11617 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build ) mkdir -p webui-generated /tmp/ui-gen cmake -DUI_SOURCE_DIR=/tmp/lc/tools/ui -DUI_BINARY_DIR=/tmp/ui-gen \ @@ -1146,7 +1146,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b11612`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b11617`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1376,12 +1376,12 @@ Current patches: | `0007-server-attach-http-frontend.patch` | **Adds `llama_server_attach(argc, argv, server_context&)`** so the `NativeServer` *attach mode* can serve an **already-loaded `LlamaModel`** over the upstream HTTP frontend — no second model load, no `start_loop()`; the LlamaModel's worker keeps driving the shared `server_context` and the HTTP routes post tasks to its queue (the queue is the synchronization point). Mechanically: (1) extracts the **pure core route table** (`health` … `slots`) out of `llama_server()` into `static void llama_server_register_common_routes(ctx_http, routes)` (shared, so the two entry points cannot drift on the core endpoint set). **Scope note (narrowed at the b10154 bump):** the helper deliberately carries **only** the stable, state-independent route table — **not** the resumable-streaming routes (their handlers differ between router / non-router), the GCP-compat shim, or the experimental **CORS-proxy / MCP-server / built-in-tools** wiring. b10154 (upstream MCP-server support) moved the streaming routes into the middle of that block and coupled tools/CORS to a per-call `server_mcp mcp_mgr` lifecycle, so the earlier contiguous "route-table + CORS-proxy + tools" extraction is no longer possible; `llama_server()` keeps all of that inline, **byte-identical to upstream b10154** (only the route-table block is factored out). (2) adds `llama_server_attach`, which parses only the HTTP-side argv via `common_params_parse`, starts the stream-session GC + `server_http_context`, registers the common route table, the **non-router** resumable-streaming handlers (upstream b10154 paths `/v1/stream` GET/DEL + `/v1/streams/lookup` POST), the GCP-compat shim, and **403 "disabled" stubs for `/cors-proxy` + `/tools`** (attach mode does not wire the experimental CORS-proxy / MCP / built-in-tools host — those belong to a full `llama-server`, not an embedded model), marks ready immediately (model already loaded), and blocks on the HTTP thread until `llama_server_request_shutdown()` — never calling `common_init()`, backend init, `ctx_server.terminate()` or `llama_backend_free()` (the embedding caller owns those). Applies after `0001`+`0006` (same file); closes the "NativeServer — reuse an already-loaded LlamaModel" TODO. Upstream-submittable ("server: let embedding callers attach the HTTP frontend to an existing server_context"). **Refreshed at the b10519 bump:** upstream #26347 dropped the API key from the `/models` + `/v1/models` public-endpoint set and deleted the two trailing `// public endpoint (no API key check)` comments on those route registrations. Those two lines sit inside this patch's route-table removal block, so `git apply` failed ("patch does not apply", `server.cpp:258`) at **every** tag from b10519 on; the fix was to drop the now-wrong comment from all four affected lines (2 on the `-` side, 2 in the extracted helper on the `+` side), keeping the helper byte-identical to the block it replaces. **This is the invariant to re-check on every bump:** the `+` side of `llama_server_register_common_routes()` must stay a verbatim copy of the route table it factors out of `llama_server()`. **Refreshed at the b11104 bump** (upstream #28690, multi-address `--host`): `server_http_context` lost its single `thread` and `listening_address` members in favour of `join()` and a `listening_addresses` vector, one listener thread per bound address. The patch still *applied* cleanly there — only its own `+` lines named the removed members — so the applier could not see it; `llama_server_attach` now logs every address and blocks in `ctx_http.join()`, exactly as upstream's `llama_server()` does. **Refreshed at the b11361 bump** (#29818 adds `POST /v1/systemone` to the route table: one line, added on both sides so the helper stays verbatim) **and at b11401** (#29895 adds a `server_child &` overload of `llama_server()` beside the declarations and a `server_child child;` at the top of the argv entry point, the context of `0007`'s first two hunks; only context moved, the `+`/`-` lines are unchanged). **Changed in 5.2.x, not a refresh: `llama_server_attach(argc, argv, server_context &, server_routes &)` serves the caller's `server_routes`** (`jllama_context::routes`, the one the model load builds) **instead of constructing one on its own frame.** `server_routes`' constructor registers a sleeping-state callback on the model's task queue, `server_queue::on_sleeping_state` only appends, so the frame-local object left a callback into freed memory behind when the attach returned, and the model's next idle sleep (`--sleep-idle-seconds`) ran it -- undefined behaviour, found at the b11361 bump and pinned by `NativeServerAttachSleepIntegrationTest` (model-backed, CI). Consequence: the route handlers read the **model's** params, so the endpoint toggles (`--metrics`, `--props`, `--slots`), `--slot-save-path`, the slot count and the sampling defaults are `ModelParameters` now (`enableMetricsEndpoint()` & co., added for that), and the attach argv configures the HTTP side only. | | `0008-server-models-worker-cmd-override.patch` | **Makes router mode usable in-JVM.** The router (`server-models.cpp`) spawns each model worker by re-executing its own binary (`get_server_exec_path()` = `/proc/self/exe` & friends) — inside a JVM that binary is `java`, not a llama-server, so embedded router workers could never start. The patch adds env `LLAMA_SERVER_WORKER_CMD` (whitespace-split; read in `server_model_meta::update_args`) which replaces only the leading binary-path token of the rendered worker args, letting an embedding host relaunch workers through its own bootstrap — e.g. `java -cp app.jar net.ladenthin.llama.server.NativeServer` (each worker is then a fresh JVM running the classic single-model `NativeServer`). Exposed in Java as `NativeServer.setWorkerCommand(String...)` (JNI `setenv`); exercised by `RouterModeIntegrationTest` (Linux CI). Upstream-submittable (also useful for containerized/wrapped deployments). **Refreshed at the b11429 bump:** #29987 (untagged `4d60b4d08`, the commit before b11429) moved `server_model_meta::update_args` and changed the function after it (`update_caps(const common_params &)`), the hunk's trailing context; the `+` lines are unchanged. Since b11401 (#29895) a worker is a router *child* whose stdout carries the state commands, which is why `LlamaLoader` prints its two diagnostic lines to stderr (a JVM worker's early `System.out` line tripped the router until 2026-10-09, `e9758054`). | | `0006-server-embed-native-server-jni.patch` | **Makes `server.cpp`'s `llama_server` embeddable in the JVM** so the `NativeServer` JNI bridge can run the full upstream HTTP server (WebUI included) inside `libjllama` — see "Two server modes" below. b9870 already exposes `int llama_server(int, char**)` (non-static; no `main` in the file), so the patch only adds embedded-mode support: (1) a `g_llama_server_embedded` flag + `llama_server_set_embedded()` / `llama_server_request_shutdown()` (declared in the committed `src/main/cpp/native_server_bridge.h`); (2) skips installing the process-wide SIGINT/SIGTERM handlers when embedded (they would hijack the JVM's); (3) in embedded mode parses the **forwarded** argv via `common_params_parse` instead of `common_params_parse_main` (whose `GetCommandLineW` recovery would pick up `java.exe`'s command line — the same Windows class of bug `0001` fixes). `llama_server_request_shutdown()` mirrors the SIGTERM path (invokes the installed `shutdown_handler` → `ctx_server.terminate()` unblocks `start_loop()`), giving JNI an out-of-band stop since `ctx_server` is loop-local. **The handler is guarded, and both guards are load-bearing:** upstream's `shutdown_handler` lambdas capture `llama_server()`'s locals (`ctx_http`, `models_routes`, `mcp_mgr`, `ctx_server`) by reference and are never cleared, while `native_server.cpp` signals *repeatedly* until the worker reports it finished (a stop issued before the handler is installed, or before `start_loop()` begins, would otherwise be lost). A request landing after `llama_server()` returned therefore ran the lambda over destroyed objects — a `SIGSEGV` in `server_http_context::stop()` in `RouterModeIntegrationTest.tearDown` (run 36485159455), with router mode's `clean_up()` (unloading workers) making the window wide. The patch (1) takes `g_shutdown_handler_mutex` for every read, write and invocation of the handler (the caller is on another thread) and (2) declares a `llama_server_shutdown_handler_reset` **after** the captured locals, so the handler is cleared before any of them is destroyed; because invocation holds the same lock, that reset waits for an invocation in progress. `0007`'s `llama_server_attach` carries the same guard. Runnable guard: `src/test/cpp/test_native_server_shutdown.cpp`. Applies **after `0001`** (which flips this call site to `common_params_parse_main`), so its context is the post-`0001` tree; regenerate against `0001`+source on a bump. Only touches `tools/server/server.cpp`. | -| `0012-model-guard-zero-split-sum-and-name-the-device-index.patch` | **A GPU that reports zero free memory makes every model load fail with the unactionable `error loading model: vector`.** `llama_model_base::load_tensors` (`src/llama-model.cpp`) weights the per-device layer split by `ggml_backend_dev_memory()`'s `free`, then normalises: `splits[i] /= split_sum`. With a single device reporting `free == 0` that is `0/0` → **NaN** in every split point; NaN compares false against everything, so the `std::upper_bound` below returns the end iterator, `layer_gpu == n_devices()`, and `devices.at(layer_gpu)` throws `std::out_of_range` — whose libc++ `what()` is the bare string `"vector"`, which `llama.cpp`'s `catch (const std::exception &)` prints verbatim. Upstream's `free == 0 && total == 0` host-memory fallback does **not** fire, because `total` is `recommendedMaxWorkingSetSize` and is non-zero. **Reachable since b10618..b10797**: upstream `8c0b9cd04` ("metal : fix memory query under low-memory conditions", [#27701](https://github.com/ggml-org/llama.cpp/pull/27701)) changed `ggml-metal-device.m` to `*free = *total > cur ? *total - cur : 0`; before that clamp an over-committed device (`currentAllocatedSize > recommendedMaxWorkingSetSize`) *underflowed* to a huge `size_t`, which normalised fine, so the same precondition was harmless. That is why the `Java Tests macOS …` jobs went red at the b10792→b10797 step while every Linux/Windows job stayed green — **and why only a GPU build can fail this way at all**: `act_gpu_layers` is `devices.empty() ? 0 : …`, so with no GPU backend `devices` is empty, every layer returns early on `cpu_dev`, and the `.at()` line is unreachable. **Shape:** the two blocks are lifted out of `load_tensors` into free functions declared in `src/llama-model.h`, purely so they can be driven by a test — the failing state needs a real over-committed GPU and cannot be arranged through any public API. `llama_model_splits_normalize()` carries **the fix**: on `split_sum == 0` it `LLAMA_LOG_WARN`s and falls back to an even split (`splits[i] = float(i+1)/splits.size()`), the only neutral choice when no device can be preferred and exactly right for a single device. `llama_model_splits_select_device()` carries **the diagnostic**: it bounds-checks the index and throws a `std::runtime_error` naming the function, the offloaded layer, the device index, the split-point count **and the split points themselves** — with NaN splits that message prints `nan` and names the cause outright, which is precisely what was missing when this had to be diagnosed by reading source. **A second, backend-independent trigger reaches the same line**, found while writing this up and verified against the unfixed library: `--tensor-split` values are parsed with `std::stof` and never range-checked (`common/arg.cpp`), so `-ts 1,-1` cancels out, `split_sum` is 0 again, the split points become `[inf, -nan]`, and every layer maps one past the last device — on CUDA, Vulkan or ROCm just as much as on Metal, with no memory pressure involved. That is what makes this an ordinary upstream defect rather than a Metal edge case, and the warning names both causes rather than only the memory one. Also adds upstream `tests/test-model-split.cpp` (5 cases in upstream's `testing.h` style) + its `llama_build_and_test` registration. Touches `src/llama-model.{cpp,h}`, `tests/test-model-split.cpp` and `tests/CMakeLists.txt` — **none** of which any other patch touches, so it is independent of all of them. **Refreshed at the b11570 bump** (#30299 links each upstream test against only the libraries it uses: `llama_build_and_test` takes `LINK `, and the registration this patch adds reads `llama_build_and_test(test-model-split.cpp LINK llama)` now -- the carried test includes `../src/llama-model.h` and needs `llama` only, like the other internal-function tests in that block; the `src/llama-model.{cpp,h}` and `tests/test-model-split.cpp` hunks are unchanged, only their offsets moved). Upstream-submittable ("model: fall back to an even split when no device reports free memory"); **not yet filed upstream**. **Runnable guard: `src/test/cpp/test_model_split.cpp`** — a FetchContent subproject builds with `LLAMA_BUILD_TESTS=OFF`, so the upstream test above is applied-but-never-compiled here (same as `0001`'s test). That file drives the same two functions from `jllama_test`, which runs on **every** platform in `C++ Tests`, so a bump that drops this patch fails the build at link time everywhere instead of surfacing as one red macOS Java job. **Verification limit — read before assuming this can be dropped:** the *failing path* still cannot be reached without a GPU backend, so the guard pins the arithmetic (what actually broke), not the end-to-end load; the end-to-end proof is the macOS CI job. On a bump, re-check whether upstream added its own `split_sum == 0` guard (grep `split_sum` in `src/llama-model.cpp`) and **drop this patch rather than refreshing it** if they did — the fail-loud applier detects "does not apply", never "upstream already fixed this". **Upstream fixed the defect on 2026-10-14, one tag after this pin:** [ggml-org/llama.cpp#29677](https://github.com/ggml-org/llama.cpp/pull/29677), commit [`df18bee11`](https://github.com/ggml-org/llama.cpp/commit/df18bee11e96557b07aa7b12db0976f3c7ac9165) ("model: split layers evenly when no device reports free memory"), one line in `load_tensors()`: `splits[i] = split_sum != 0.0f ? splits[i] / split_sum : float(i + 1) / n_devices()` -- the same even-split fallback as `llama_model_splits_normalize()` here, without the warning, without the `select_device` diagnostic and **without a test** (the PR touches `src/llama-model.cpp` only). It is **not in b11612** (`10da8c463`, 15 commits before it on master; `src/llama-model.cpp:1648` there is still the bare `splits[i] /= split_sum`, measured), which is why the patch is still carried at this pin and applies unchanged. **At the first bump past it the drop-check above fires: drop `0012`, do not refresh it.** What that does to the guard is decided in `TODO.md` ("Drop patch `0012` at the first bump past b11612"): `test_model_split.cpp` links the two free functions only this patch adds, so it cannot outlive the patch as it is, and upstream shipped no test to point it at. | +| `0012-model-guard-zero-split-sum-and-name-the-device-index.patch` | **A GPU that reports zero free memory makes every model load fail with the unactionable `error loading model: vector`.** `llama_model_base::load_tensors` (`src/llama-model.cpp`) weights the per-device layer split by `ggml_backend_dev_memory()`'s `free`, then normalises: `splits[i] /= split_sum`. With a single device reporting `free == 0` that is `0/0` → **NaN** in every split point; NaN compares false against everything, so the `std::upper_bound` below returns the end iterator, `layer_gpu == n_devices()`, and `devices.at(layer_gpu)` throws `std::out_of_range` — whose libc++ `what()` is the bare string `"vector"`, which `llama.cpp`'s `catch (const std::exception &)` prints verbatim. Upstream's `free == 0 && total == 0` host-memory fallback does **not** fire, because `total` is `recommendedMaxWorkingSetSize` and is non-zero. **Reachable since b10618..b10797**: upstream `8c0b9cd04` ("metal : fix memory query under low-memory conditions", [#27701](https://github.com/ggml-org/llama.cpp/pull/27701)) changed `ggml-metal-device.m` to `*free = *total > cur ? *total - cur : 0`; before that clamp an over-committed device (`currentAllocatedSize > recommendedMaxWorkingSetSize`) *underflowed* to a huge `size_t`, which normalised fine, so the same precondition was harmless. That is why the `Java Tests macOS …` jobs went red at the b10792→b10797 step while every Linux/Windows job stayed green — **and why only a GPU build can fail this way at all**: `act_gpu_layers` is `devices.empty() ? 0 : …`, so with no GPU backend `devices` is empty, every layer returns early on `cpu_dev`, and the `.at()` line is unreachable. **Shape:** the two blocks are lifted out of `load_tensors` into free functions declared in `src/llama-model.h`, purely so they can be driven by a test — the failing state needs a real over-committed GPU and cannot be arranged through any public API. `llama_model_splits_normalize()` carries **the fix**: on `split_sum == 0` it `LLAMA_LOG_WARN`s and falls back to an even split (`splits[i] = float(i+1)/splits.size()`), the only neutral choice when no device can be preferred and exactly right for a single device. `llama_model_splits_select_device()` carries **the diagnostic**: it bounds-checks the index and throws a `std::runtime_error` naming the function, the offloaded layer, the device index, the split-point count **and the split points themselves** — with NaN splits that message prints `nan` and names the cause outright, which is precisely what was missing when this had to be diagnosed by reading source. **A second, backend-independent trigger reaches the same line**, found while writing this up and verified against the unfixed library: `--tensor-split` values are parsed with `std::stof` and never range-checked (`common/arg.cpp`), so `-ts 1,-1` cancels out, `split_sum` is 0 again, the split points become `[inf, -nan]`, and every layer maps one past the last device — on CUDA, Vulkan or ROCm just as much as on Metal, with no memory pressure involved. That is what makes this an ordinary upstream defect rather than a Metal edge case, and the warning names both causes rather than only the memory one. Also adds upstream `tests/test-model-split.cpp` (5 cases in upstream's `testing.h` style) + its `llama_build_and_test` registration. Touches `src/llama-model.{cpp,h}`, `tests/test-model-split.cpp` and `tests/CMakeLists.txt` — **none** of which any other patch touches, so it is independent of all of them. **Refreshed at the b11570 bump** (#30299 links each upstream test against only the libraries it uses: `llama_build_and_test` takes `LINK `, and the registration this patch adds reads `llama_build_and_test(test-model-split.cpp LINK llama)` now -- the carried test includes `../src/llama-model.h` and needs `llama` only, like the other internal-function tests in that block; the `src/llama-model.{cpp,h}` and `tests/test-model-split.cpp` hunks are unchanged, only their offsets moved). Upstream-submittable ("model: fall back to an even split when no device reports free memory"); **not yet filed upstream**. **Runnable guard: `src/test/cpp/test_model_split.cpp`** — a FetchContent subproject builds with `LLAMA_BUILD_TESTS=OFF`, so the upstream test above is applied-but-never-compiled here (same as `0001`'s test). That file drives the same two functions from `jllama_test`, which runs on **every** platform in `C++ Tests`, so a bump that drops this patch fails the build at link time everywhere instead of surfacing as one red macOS Java job. **Verification limit — read before assuming this can be dropped:** the *failing path* still cannot be reached without a GPU backend, so the guard pins the arithmetic (what actually broke), not the end-to-end load; the end-to-end proof is the macOS CI job. On a bump, re-check whether upstream added its own `split_sum == 0` guard (grep `split_sum` in `src/llama-model.cpp`) and **drop this patch rather than refreshing it** if they did — the fail-loud applier detects "does not apply", never "upstream already fixed this". **Upstream fixed the defect on 2026-10-14, one tag after this pin:** [ggml-org/llama.cpp#29677](https://github.com/ggml-org/llama.cpp/pull/29677), commit [`df18bee11`](https://github.com/ggml-org/llama.cpp/commit/df18bee11e96557b07aa7b12db0976f3c7ac9165) ("model: split layers evenly when no device reports free memory"), one line in `load_tensors()`: `splits[i] = split_sum != 0.0f ? splits[i] / split_sum : float(i + 1) / n_devices()` -- the same even-split fallback as `llama_model_splits_normalize()` here, without the warning, without the `select_device` diagnostic and **without a test** (the PR touches `src/llama-model.cpp` only). It is **not in b11612** (`10da8c463`, 15 commits before it on master; `src/llama-model.cpp:1648` there is still the bare `splits[i] /= split_sum`, measured), which is why the patch is still carried at this pin and applies unchanged. **At the first bump to a tag that carries `df18bee11` (none up to b11618; b11617 still has the bare division at `src/llama-model.cpp:1648`) the drop-check above fires: drop `0012`, do not refresh it.** What that does to the guard is decided in `TODO.md` ("Drop patch `0012` at the first bump that carries #29677"): `test_model_split.cpp` links the two free functions only this patch adds, so it cannot outlive the patch as it is, and upstream shipped no test to point it at. | | `0014-common-log-callback-sink.patch` | **Gives `common_log` a callback sink: `common_log_set_callback(log, cb, user_data)` (`common/log.{h,cpp}`).** This is what `LlamaModel.setLogger` hooks. Before it, the Java logger was a `llama_log_set()` callback, which has two holes, both found while chasing `slot print_timing` lines interleaving with the Atmosphere agent's streamed answer: **(a)** every model load runs `common_init()`, which re-points `llama_log_set()` at `common_log_default_callback` (`common.cpp:394`), so `setLogger(…)` *before* `new LlamaModel(…)` silently lost the callback; **(b)** the server's own `SRV_*`/`SLT_*` macros are `LOG_INF` and write straight into `common_log`, which `llama_log_set()` never carried, so the per-request `slot …`/`srv …` lines could not be routed to Java at all (the reason `LlamaModelTest#testLogText/JSON` sat `@Disabled` for years). `common_log` upstream offers file, colors, prefix, timestamps, verbosity and JSONL but no hook. The patch adds one: while a callback is set, the worker thread hands every entry to it **instead of** printing to stdout/stderr (a `--log-file` still receives them); the callback gets the bare formatted message (no prefix/timestamp/colors) with the `ggml_log_callback` signature; swapping the callback pauses the worker first, so queued entries reach the *previous* sink (which is what makes `setLogger(format, null)` a synchronous drain). With the sink, `common_init()`'s `llama_log_set()` reset is harmless — it points at the default callback that feeds `common_log`, i.e. exactly the path into the sink — so the ordering problem (a) disappears without any re-install logic, and the `srv`/`slot` lines (b) arrive because they are `common_log` entries. `jllama.cpp`'s `setLogger` therefore sets `common_log_set_callback(common_log_main(), trampoline)` **plus** `llama_log_set(common_log_default_callback)` (so llama/ggml lines feed `common_log` even before the first load). Two consequences to know: the callback runs on `common_log`'s **worker thread**, a plain `std::thread` llama.cpp re-creates on every pause/resume and never attaches to the JVM — the trampoline attaches it **once per worker thread** as a daemon, named `jllama-log-worker`, and a `thread_local` guard detaches it when the thread exits (`get_jni_env_attaching`; `common_log::pause()` joins the worker, a normal exit on which TLS destructors run, and `JNI_OnUnload` pauses it before anything else). Until 5.2.x it attached and detached per line, which created a `java.lang.Thread` object per log line (13 lines of a failed load, 13 `Thread` objects -- `LlamaLoggerTest` pins the single attach now). Each call runs inside a `PushLocalFrame`/`PopLocalFrame` pair, because a thread that stays attached and has no Java frame would otherwise accumulate local references, `setLogger` must call `common_log_set_callback` **outside** `g_log_mutex`, because the pause joins the worker, which needs that mutex to read the callback, and `setLogger` callers are serialized by a **separate** `g_set_logger_mutex`: two unserialized swaps race on the worker's `std::thread` (one joins it while the other assigns a fresh thread over the still-joinable object = `std::terminate`, the whole JVM), which `LlamaLoggerTest#concurrentSetLoggerCallsDoNotRaceOnTheLogWorker` reproduced before the mutex existed. Two caveats the Javadoc carries: a caller must not hold a lock the *previous* callback needs (the drain runs it on the worker while the caller waits), and the verbosity threshold is process-wide and reset by **every** load (`common_params_parse` ends with `common_log_set_verbosity_thold(params.verbosity)`, default 3), so a load without `-lv` puts it back to 3 — a review assumed the opposite, and `LlamaLoggerTest#verbosityThresholdIsProcessWideAndEveryLoadSetsIt` now pins the measured behaviour. And the verbosity threshold applies *before* the sink: at the default (`3`) the Java logger sees errors, warnings and the server's INFO lines, while llama/ggml INFO lines (`common_log_get_verbosity` maps them to TRACE = 4) arrive only from `setLogVerbosity(4)` on — the same filtering the console gets, and a behaviour change for consumers who captured the unfiltered `llama_log_set()` stream before. **Runnable guards:** `src/test/cpp/test_common_log_callback.cpp` (6 tests over a private `common_log_init()` instance: delivery, bare text under prefix+timestamps, clear, swap-drains-to-old-sink, file kept, levels pass through) links the function on every platform, so a bump that drops the patch reds `C++ Tests` at link time; `LlamaLoggerTest` (model-free, needs only `libjllama`: a logger set before a deliberately failing load on a non-GGUF file sees the `srv … loading model` INFO line and llama's ERROR line, in TEXT and JSON) and the re-enabled `LlamaModelTest#testLogText/testLogJSON` plus `#testLoggerSetBeforeLoadSurvivesTheLoad` (vocab-only load) cover the Java side. Upstream-submittable ("common : add a callback sink to common_log for embedding hosts"); **not yet filed upstream**. Touches only `common/log.{h,cpp}`, which no other patch touches. **Refreshed at the b11401 bump** (#29895 adds a `colors` member next to the sink's members and a `get_colors()` next to `set_callback()`; only context moved, the `+`/`-` lines are unchanged, and upstream still has no hook -- it added `common_log_get_colors` only). **Refreshed at the b11612 bump** (`common_log_set_stdo()`, a stdout/stderr on-off switch stored as a `stdo` member right where the sink's members sit, and `print_entry()` now prints to the console only `if (stdo)`): three hunks of `common/log.cpp` lost their context and were regenerated; the sink keeps precedence -- `if (callback) { callback(...) } else if (stdo) { e.print(); }` -- so a sink still replaces the console and `set_stdo(false)` without a sink still silences it. Still no hook upstream (`common/log.h` names only `common_log_default_callback`). **On a bump, check whether upstream added a hook of its own (grep `callback` in `common/log.h`) and, if so, DROP this patch and port `setLogger` to theirs rather than refreshing it.** | | `0015-rpc-embeddable-client-and-stoppable-server.patch` | **Makes ggml-rpc usable inside a JVM** (see "RPC backend" below). Upstream treats every client-side RPC problem as `GGML_ABORT` — a malformed endpoint, a server that is not running, a failed handshake — which in a JVM kills the application over a typo in `--rpc`. **(1)** The *registration* path reports failure instead: `rpc_dispatcher::start()` returns `false`, `try_get_dispatcher()` `nullptr`, `ggml_backend_rpc_get_device_count()` `0`, `ggml_backend_rpc_add_server()` `nullptr`, and `add_rpc_devices()` (`common/arg.cpp`) throws `std::invalid_argument` naming the server where it used to register `nullptr` and silently run without it. Every path *after* registration keeps the original contract through `get_dispatcher()`, which still aborts — a server that vanishes mid-inference stays fatal (TODO). **(2)** `add_server()`'s cache hit is re-checked (a stopped server used to pass registration and abort on the first tensor upload), and `ggml_backend_rpc_get_device_memory()` of a gone server returns 0/0 instead of aborting — RPC devices stay in ggml's process-wide registry forever, and `common_init()` queries the memory of **every** registered device at `-lv 4`, so a stopped server would otherwise take down an unrelated later load. **(3)** The server becomes stoppable: `ggml_backend_rpc_stop_server()` (disconnects the client being served, wakes `accept()` with a loopback connection — the one portable way) and `ggml_backend_rpc_server_listening()`; the loop now reaches its cleanup and frees its backends on every early return, but deliberately **not** `rpc_transport_shutdown()`, whose `WSACleanup()` would invalidate every other RPC socket in the process. **(4)** transport: `MSG_NOSIGNAL`/`SO_NOSIGPIPE`, `socket_t::shutdown()`, and the fds `connect()`/`create_server()`/`accept()` leaked on their error paths are closed. Touches `ggml/include/ggml-rpc.h`, `ggml/src/ggml-rpc/{ggml-rpc.cpp,transport.cpp,transport.h}` and one function in `common/arg.cpp` (`add_rpc_devices`, away from `0001`'s `common_params_parse` hunks). **Runnable guard: `src/test/cpp/test_rpc.cpp`** — links both new functions (a bump that drops the patch fails `C++ Tests` at link time on every platform) and drives registration, stop and the memory query over loopback. Upstream-submittable; **not yet filed upstream**. On a bump, check whether upstream grew its own stop function or non-aborting registration (grep `stop_server` / `GGML_ABORT("Failed to connect` in `ggml-rpc.cpp`). **Refreshed at the b11450 bump** (#26610, `-sm tensor` over RPC, ~800 lines in `ggml-rpc.cpp`, protocol 7 → 8): three hunks lost their context -- new dispatcher members beside `start()`, and new `get_proc_address` names before `return NULL` (the two new names now sit after `ggml_backend_comm_allreduce_tensor`); the `+`/`-` lines are unchanged. The server-to-server comm #26610 adds (a listener on `0.0.0.0`, an `accept()` a stop does not wake) is on file in `TODO.md`. **Refreshed at the b11558 bump** (#29544 turns `GGML_RPC_DEBUG` into a verbosity level 0-3: one `ggml-rpc/log.h` with `LOG_ERROR`/`LOG_WARN`/`LOG_INFO` and `LOG_DBG`/`LOG_DBG2`/`LOG_DBG3`, every `GGML_LOG_*`, `printf` and `fprintf(stderr)` of the backend routed through it, and `transport_name()` where `shutdown()` goes in `transport.h`): 12 of the 24 hunks lost their context and were regenerated against the new text; the changes are the same, and the patch's own five error lines (the HELLO handshake in `negotiate_hello`, four in `rpc_dispatcher::start()`) use `LOG_ERROR` now like the code around them (`LOG_ERROR` is `GGML_LOG_ERROR`). One consequence outside the patch: the server's `Accepted client connection` line is `LOG_INFO` through ggml's log callback now (stderr by default, a `setLogger` sink at verbosity 4), no longer `printf` to stdout; `smoke-rpc.sh` captures both streams into `rpc-server.log`, so its grep is unaffected. | | `0017-ggml-cpu-x86-prefetch-byte-offset-and-msvc-type.patch` | **Makes `GGML_CPU_ALL_VARIANTS=ON` buildable with plain clang on Windows, and fixes a prefetch address that is wrong everywhere.** The four `_mm_prefetch` calls in `ggml/src/ggml-cpu/arch/x86/quants.c` (the SSSE3-without-AVX branch of `ggml_vec_dot_q4_0_q8_0`, and the only `_mm_prefetch` calls in the whole ggml tree) carry two independent defects; both are fixed by routing the address through one new `static inline` helper in the new header `ggml/src/ggml-cpu/arch/x86/prefetch.h`, which is included relatively, so **no CMake change is needed**. **(1) Type -- fatal only on a current clang:** MSVC declares the intrinsic as `void _mm_prefetch(char const *, int)`; clang's own `xmmintrin.h` defines a casting macro but only `#ifndef _MSC_VER`, so on a `*-windows-msvc` target clang leaves the declaration to Microsoft and a typed pointer is `-Wincompatible-pointer-types`. **clang 22 promoted that to an error by default** ([llvm-project #157364](https://github.com/llvm/llvm-project/pull/157364); the [22.1.0 release notes](https://releases.llvm.org/22.1.0/tools/clang/docs/ReleaseNotes.html) add that `-Wno-error=incompatible-pointer-types` downgrades it) -- *not* clang 16, which promoted only `-Wincompatible-FUNCTION-pointer-types`; an earlier version of this row said otherwise. Measured on the real file at `-msse4.2`, unpatched → patched: clang **20.1.8** 4 warnings → 0/0, clang **22.1.3** 4 errors → 0/0, clang **23.1.3** 4 errors → 0/0. Plain clang is exactly what `cmake/x64-windows-llvm.cmake` selects -- and plain clang is the only way to get all **14** CPU variants, because `clang-cl` makes CMake's `MSVC` true and the `if (NOT MSVC)` block in `ggml/src/CMakeLists.txt` then drops `zen4`, `cooperlake`, `sapphirerapids`, `ivybridge` and `piledriver`. **(2) Arithmetic:** `&x[ib] + sizeof(block_q4_0)` adds `sizeof()` *elements*, not bytes -- +324 and +1156 instead of +18 and +34 (the sizes `ggml-common.h` pins by `static_assert`), and +648 instead of +36 for the `2 *` pair. That half is wrong on every platform, including where the call compiles. **Measured:** no throughput difference (see the test-table row), so this is a correctness and portability fix, not a performance one. **Upstream-submittable and meant to be submitted.** **Why upstream's own build is green is measured, not an open question:** their `windows-cpu` release job builds exactly this with plain clang, `GGML_CPU_ALL_VARIANTS=ON` and no suppressing flag (`CMAKE_ARGS` carries only build-target toggles), and ships `ggml-cpu-sse42.dll` -- because the `windows-2025-vs2026` image has **LLVM 20.1.8** on `PATH`, where these are warnings. That is narrower than it looks: the same image also ships **clang 22.1.3** inside Visual Studio 2026 (`VC/Tools/Llvm/x64`), which *fails* on the unpatched file -- upstream reaches the working compiler only because `C:\Program Files/LLVM/bin` comes first on `PATH`. It is **not** a header difference: all three MSVC toolsets on that image (14.29.30133, 14.44.35207, 14.51.36231) declare it identically as `extern void _mm_prefetch(char const*_A, int _Sel);` -- that was the first hypothesis, and a one-off probe workflow (deleted again once it had answered) disproved it by printing the declaration from each toolset on the image. So this patch is **ahead of** upstream rather than diverging: their build breaks the moment the image ships a clang that makes the diagnostic fatal, and until then they compile four warnings and a prefetch address 18x/34x too far. **Runnable guard: `src/test/cpp/test_prefetch.cpp`.** On a bump, run `git grep -n _mm_prefetch ggml/src/ggml-cpu/arch/x86/quants.c` at the new tag: if the call sites already cast **before** the addition, **drop this patch rather than refreshing it**. | | `0016-model-kolibri1.patch` | **Adds Aleph Alpha's Kolibri-1 (architecture `kolibri1`) ahead of upstream** -- a 78B German/English reasoning MoE (3.46B active) that upstream does not support yet ([ggml-org/llama.cpp#29922](https://github.com/ggml-org/llama.cpp/issues/29922), no PR at the time). A **temporary carry, not a fix**: drop it the moment upstream registers the architecture (`git grep -n kolibri src/llama-arch.cpp` at the new tag), and keep `test_kolibri1.cpp` -- its numerical comparisons must then pass against upstream's implementation, while its GGUF-format rows follow whatever format upstream's converter fixes (see the test-table row). **Architecture** (Aleph Alpha's vLLM reference, `aleph_alpha_inference/kolibri1.py` at `049a6a7`): GQA with per-head q/k RMSNorm; `layer_types` interleaves sliding-window layers (NEOX RoPE) with full-attention layers that have **no** positional encoding; sandwich norms around attention and MoE; every layer routed MoE + one ungated shared expert; the router selects top-k on **`logits + expert_bias`** and weights by the **unbiased `sigmoid(logits)`** -- not DeepSeek-V3's sigmoid router (selection on `sigmoid(logits) + bias`), which picks other experts as soon as the bias is non-zero. The graph therefore builds the selection itself and passes it to `build_moe_ffn` (`selected_experts_in`), leaving upstream's shared MoE code untouched. **Derived from two community ports, which are incompatible with each other's GGUFs** -- Eliasfpv28's AFMoE-based one (base b11378, `expert_gating_func` 2, pre-tokenizer `qwen2`) and the Qwen3-MoE-based one behind Hob-forge's GGUFs (base b11381, as carried in [mjwsolo/localcode#101](https://github.com/mjwsolo/localcode/pull/101): gating 5, pre-tokenizer `kolibri1`, rejects gating 2). This patch **loads both dialects**: gating 2, 5 or absent (all mean the one router), `kolibri1` mapped to the Qwen2 pre-tokenizer in `llama-vocab.cpp`, `expert_weights_norm` / `expert_shared_count` / `output.weight` optional. Every source, with license, is listed in the patch header; `REUSE.toml` annotates the file `MIT AND Apache-2.0` (Eliasfpv28's additions are Apache-2.0). Touches `src/llama-arch.{h,cpp}`, `src/llama-model.cpp` (mapping + NEOX rope list), `src/llama-vocab.cpp` (one name), `src/models/models.h` and the new `src/models/kolibri1.cpp`; no converter. **Runnable guard: `src/test/cpp/test_kolibri1.cpp`** (writes tiny random GGUFs in both dialects and compares every logit with an independent double-precision reference). **Verified against the real 78B model on 2026-10-11** (rented 16-vCPU / 123 GB instance, both published dialects fetched on the machine): both load, and a chat turn through the model's own template answers correctly in each -- `content` `"Berlin."` for a German question and `"51"` for 17x3, with the chain of thought in `reasoning_content` and `finish_reason` `stop`. **The two dialects tokenize identically** (`[452 22090 493 1678 2459 5513 46]`, round-tripping to the input sentence), which is the direct evidence that mapping `kolibri1` to the Qwen2 pre-tokenizer is right. Q4_K_M (44 GB) gave 13.4 t/s and Q3_K_S (31 GB) 14.6 t/s of generation on 16 cores. **Two traps for whoever repeats this:** a *raw* completion on this instruct model degenerates into repetition whatever the patch does, so measure through the chat endpoint; and `chatCompleteText` returns only `content`, which is **empty** when a reasoning model spends its token budget inside the thinking block -- use `handleChatCompletions` and a budget in the hundreds. **Still not verified:** GPU backends -- the community ports report real-model checks (top-1 67/67 against a float64 reference for Q8_0, tool calls, German text). | -| `0018-ggml-amx-free-the-aligned-buffer.patch` | **The AMX weight buffer is freed with `free()` but allocated with `_aligned_malloc()` on Windows -- `STATUS_HEAP_CORRUPTION` (0xC0000374) the moment a model is unloaded.** `ggml_backend_amx_buffer_type_alloc_buffer()` (`ggml/src/ggml-cpu/amx/amx.cpp`) allocates with `ggml_aligned_malloc()`, which is `_aligned_malloc()` under `_MSC_VER` (plain clang on a `*-windows-msvc` target defines it too); `ggml_backend_amx_buffer_free_buffer()` called `free()` on it. Fine on Linux (`posix_memalign` + `free`), fatal on Windows, and only there. The patch frees with `ggml_aligned_free(buffer->context, buffer->size)`, exactly what `ggml_backend_cpu_buffer_free_buffer()` does; on Linux that is `free()`, so nothing changes there. **Reached by** the `sapphirerapids` CPU module (the only variant built with `AMX_TILE AMX_INT8`, which `ggml_backend_load_best` picks on Sapphire/Emerald/Granite Rapids; `ggml_amx_init()` is `return true` on `_WIN32`) and a model whose weights have AMX kernels (Q4_0, Q4_1, Q8_0, Q4_K, Q5_K, Q6_K, IQ4_XS -- i.e. every CI model). **Found by the second full run of the modular natives** (38069595101): `Java Tests Windows 2025 x86_64 (clang)` landed on an `Intel(R) Xeon(R) 6973P-C` runner (Granite Rapids), loaded `ggml-cpu-sapphirerapids.dll`, ran `GrammarExample`'s two generations fine and died with exit code -1073740940 at `LlamaModel.close()`; the first run (38065017595) had the same job green on an AMD EPYC 7763 (no AVX-512, `haswell`). **The Windows runners are a CPU lottery**, so which CPU module a Windows job exercises differs per run; before the modular natives the Windows jar was a static Haswell-level build and this code never ran. **Of the six callers of `ggml_aligned_malloc` in the tree this was the only mismatched pair** (`ggml-backend.cpp`, `ggml-cpu.c` twice, `ggml-openvino.cpp`, `ggml-zdnn.cpp` and `ggml.c` all pair correctly), and `ggml-backend.cpp:2464` is the line the fix copies. **Reproduced deliberately rather than waiting for the lottery again**: on a rented Granite Rapids machine, one CPU module staged at a time with the same model and the DLLs CI itself built, `sapphirerapids` crashed **6 of 6** runs -- including at **one** thread, which rules out a data race -- while `cooperlake`, `icelake` and `haswell` were clean 6 of 6, and on Linux with the same CPU all six exit 0 with byte-identical output. The crash lands *after* the inference prints its timings, in cleanup: the signature of a heap corrupted earlier and detected at `free`. With the patch applied the same matrix on the same machine is **clean in all twelve runs**. **The guard is a configure-time check, not a `jllama_test` case**: the AMX code exists only in that one variant of a `GGML_CPU_ALL_VARIANTS` build and executes only on AMX hardware, which the static test build, the local agent's Zen 3 and the cloud sandboxes lack -- so **`llama/cmake/check-aligned-free-pairing.cmake`** fails the configure on **every** platform when a ggml source calls `ggml_aligned_malloc` and never `ggml_aligned_free` (verified red against the unpatched source, naming `amx.cpp`, and green with this patch). A run-time test cannot replace it: without an AMX CPU there is nothing to observe, which is exactly how this shipped. What exercises it: the Windows x86-64 Java test job and the `windows-x86-64` smoke row whenever GitHub hands them an Intel AMX runner; the patch itself is held by `check-patches.py`, applies at b11612, and the patched file compiles with the variant's flags (GCC 13, `-mamx-tile -mamx-int8 -mavx512vnni ...`). Upstream master still carried the `free()` on 2026-10-10; **not yet filed upstream** (nothing before the release). **Drop-check on every bump:** `grep -n "free(buffer->context)" ggml/src/ggml-cpu/amx/amx.cpp` at the new tag -- gone means upstream fixed it, drop the patch. | +| `0018-ggml-amx-free-the-aligned-buffer.patch` | **The AMX weight buffer is freed with `free()` but allocated with `_aligned_malloc()` on Windows -- `STATUS_HEAP_CORRUPTION` (0xC0000374) the moment a model is unloaded.** `ggml_backend_amx_buffer_type_alloc_buffer()` (`ggml/src/ggml-cpu/amx/amx.cpp`) allocates with `ggml_aligned_malloc()`, which is `_aligned_malloc()` under `_MSC_VER` (plain clang on a `*-windows-msvc` target defines it too); `ggml_backend_amx_buffer_free_buffer()` called `free()` on it. Fine on Linux (`posix_memalign` + `free`), fatal on Windows, and only there. The patch frees with `ggml_aligned_free(buffer->context, buffer->size)`, exactly what `ggml_backend_cpu_buffer_free_buffer()` does; on Linux that is `free()`, so nothing changes there. **Reached by** the `sapphirerapids` CPU module (the only variant built with `AMX_TILE AMX_INT8`, which `ggml_backend_load_best` picks on Sapphire/Emerald/Granite Rapids; `ggml_amx_init()` is `return true` on `_WIN32`) and a model whose weights have AMX kernels (Q4_0, Q4_1, Q8_0, Q4_K, Q5_K, Q6_K, IQ4_XS -- i.e. every CI model). **Found by the second full run of the modular natives** (38069595101): `Java Tests Windows 2025 x86_64 (clang)` landed on an `Intel(R) Xeon(R) 6973P-C` runner (Granite Rapids), loaded `ggml-cpu-sapphirerapids.dll`, ran `GrammarExample`'s two generations fine and died with exit code -1073740940 at `LlamaModel.close()`; the first run (38065017595) had the same job green on an AMD EPYC 7763 (no AVX-512, `haswell`). **The Windows runners are a CPU lottery**, so which CPU module a Windows job exercises differs per run; before the modular natives the Windows jar was a static Haswell-level build and this code never ran. **Of the six callers of `ggml_aligned_malloc` in the tree this was the only mismatched pair** (`ggml-backend.cpp`, `ggml-cpu.c` twice, `ggml-openvino.cpp`, `ggml-zdnn.cpp` and `ggml.c` all pair correctly), and `ggml-backend.cpp:2464` is the line the fix copies. **Reproduced deliberately rather than waiting for the lottery again**: on a rented Granite Rapids machine, one CPU module staged at a time with the same model and the DLLs CI itself built, `sapphirerapids` crashed **6 of 6** runs -- including at **one** thread, which rules out a data race -- while `cooperlake`, `icelake` and `haswell` were clean 6 of 6, and on Linux with the same CPU all six exit 0 with byte-identical output. The crash lands *after* the inference prints its timings, in cleanup: the signature of a heap corrupted earlier and detected at `free`. With the patch applied the same matrix on the same machine is **clean in all twelve runs**. **The guard is a configure-time check, not a `jllama_test` case**: the AMX code exists only in that one variant of a `GGML_CPU_ALL_VARIANTS` build and executes only on AMX hardware, which the static test build, the local agent's Zen 3 and the cloud sandboxes lack -- so **`llama/cmake/check-aligned-free-pairing.cmake`** fails the configure on **every** platform when a ggml source calls `ggml_aligned_malloc` and never `ggml_aligned_free` (verified red against the unpatched source, naming `amx.cpp`, and green with this patch). A run-time test cannot replace it: without an AMX CPU there is nothing to observe, which is exactly how this shipped. What exercises it: the Windows x86-64 Java test job and the `windows-x86-64` smoke row whenever GitHub hands them an Intel AMX runner; the patch itself is held by `check-patches.py`, applies at b11617, and the patched file compiles with the variant's flags (GCC 13, `-mamx-tile -mamx-int8 -mavx512vnni ...`). Upstream master still carried the `free()` on 2026-10-10; **not yet filed upstream** (nothing before the release). **Drop-check on every bump:** `grep -n "free(buffer->context)" ggml/src/ggml-cpu/amx/amx.cpp` at the new tag -- gone means upstream fixed it, drop the patch. | | `0019-ggml-cpu-select-the-neon-fp32-to-fp16-conversion.patch` | **Every ARM build converts fp32 to fp16 in software, ours and upstream's.** The `__ARM_NEON` branch of `simd-mappings.h` defines both `GGML_CPU_COMPUTE_*` macros and selects only `FP16_TO_FP32`, so `GGML_CPU_FP32_TO_FP16` falls through to `GGML_COMPUTE_FP32_TO_FP16` -- ggml-base's deliberately portable software bit manipulation -- while `neon_compute_fp32_to_fp16`, one `fcvt`, stays defined and referenced by nothing. The `GGML_CPU_*` pair exists to be the *accelerated* variant; on ARM it was defined to be the unaccelerated one. `__POWER9_VECTOR__` and `__riscv_zfhmin` select both, and the comment on that fallback has always promised NEON does too. **Not a regression** -- absent in all 25 most recent commits touching the file, back to 2025-04-10. **Why only ARM feels it:** `ggml_cpu_fp32_to_fp16`, the `from_float` of `GGML_TYPE_F16`, has vector paths for F16C, RISC-V and s390x (the last from #30157) and **none** for NEON, so there the whole function is the scalar loop through the macro. **In the shipped artifacts:** no `fcvt` of any form in that function in any of this project's 17 ARM libraries (`cpu-linux-aarch64` 8 variants, `cpu-android-aarch64` 7, `cpu-windows-aarch64`, `metal-macos-aarch64`) nor in upstream's own `bin-ubuntu-arm64` / `bin-macos-arm64`; a plain `gcc` with no flags defines `__ARM_NEON` and `__ARM_FP16_FORMAT_IEEE` on aarch64, so nothing can miss it. **Measured 2.45x-13.3x** on five ARM core types, three operating systems and nine compiler builds, with the opposite direction in the same run as a control (within 3.2 % everywhere); **bit-exact over all 2^32 float inputs**, differing only in NaN payload, where x86 F16C gives the same numbers -- so it makes ARM agree with what x86 already ships. **Not an inference speed-up:** `perf` puts 83.9 % of an F16 `pp512` run in tinyBLAS's gemm, which converts in its own microkernel, and 0.03 % here. The patch also moves #30270's two F16C selections from 88 lines below the chain into the `__F16C__` branch (behaviour-preserving, proven at object level) and puts an `#error` pair in their place that refuses a branch defining a COMPUTE macro without selecting it -- its position chosen by a mutation measurement, not by taste (4 of 4 correctness cases caught before the fallback against 3 of 4 directly after the chain). Full record: [`docs/benchmarks/arm-fp16-conversion.md`](docs/benchmarks/arm-fp16-conversion.md). **Submitted upstream as [ggml-org/llama.cpp#30505](https://github.com/ggml-org/llama.cpp/pull/30505)** (open), byte-identical to this diff. **Drop-check on every bump:** `git grep -n "GGML_CPU_FP32_TO_FP16(x) GGML_CPU_COMPUTE_FP32_TO_FP16" ggml/src/ggml-cpu/simd-mappings.h` at the new tag -- if the `__ARM_NEON` branch carries that line, **drop this patch rather than refreshing it**. | **Dropped patches** (`0004`, `0005`, `0009`, `0010`, `0011`, `0013` -- upstream fixed each defect @@ -2412,7 +2412,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b11612`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b11617`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.18.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index ad375739..c854cab0 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b11612](https://img.shields.io/badge/llama.cpp-%23b11612-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b11612) +[![llama.cpp b11617](https://img.shields.io/badge/llama.cpp-%23b11617-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b11617) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/TODO.md b/TODO.md index c258d158..5f421c27 100644 --- a/TODO.md +++ b/TODO.md @@ -842,9 +842,9 @@ whose "17 issues found" status once posted on PRs #248/#298; it posts nothing on never set up deliberately. Whether the project wants one, with which policy, is the owner's call; nothing here blocks on it. -### Drop patch `0012` at the first bump past b11612 (upstream merged the fix as #29677) +### Drop patch `0012` at the first bump that carries #29677 (upstream merged the fix on 2026-10-14) -**The drop-check of `0012` fires at the *next* bump, not at the b11612 one.** On 2026-10-14 upstream +**The drop-check of `0012` fires at the first bump to a tag that contains `df18bee11` -- not at b11612, and not at b11617 either** (checked: no tag up to b11618 carries it; b11617 still divides by `split_sum` unguarded at `src/llama-model.cpp:1648`). On 2026-10-14 upstream merged [ggml-org/llama.cpp#29677](https://github.com/ggml-org/llama.cpp/pull/29677), commit [`df18bee11`](https://github.com/ggml-org/llama.cpp/commit/df18bee11e96557b07aa7b12db0976f3c7ac9165) ("model: split layers evenly when no device reports free memory"), one line in `load_tensors()`: @@ -945,7 +945,7 @@ accepted upstream change deletes a patch from the bump checklist once the pin is direction), `0002` (preserve the caller's load-progress callback), `0006` (embeddable `llama_server`), `0007` (`llama_server_attach`), `0008` (`LLAMA_SERVER_WORKER_CMD`), ~~`0012`~~ (zero split-sum guard -- **fixed upstream independently on 2026-10-14 by #29677**, nothing left to submit; - dropped at the first bump past b11612, see the section right above), `0014` (`common_log_set_callback`), `0015` + dropped at the first bump that carries #29677, see the section right above), `0014` (`common_log_set_callback`), `0015` (non-aborting RPC registration, stoppable server, transport fixes), `0017` (x86 prefetch helper), **`0018`** (the AMX buffer released with `free()` instead of `ggml_aligned_free` -- one line, with the 6/6-against-0/6 measurement in its header and a configure-time guard that is red without it; diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index cd69e3c5..ab01cfb9 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -887,4 +887,6 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b11573–b11574 | 1 commit, 10 files, ~822/128 lines (58 KiB, the target of the series; no `tools/ui` change). **`ggml/src/ggml-hexagon/`** (#30341 "hexagon: optimize CPY/CONCAT — round 3, short-rows edition"): the Qualcomm Hexagon HTP backend's copy and concat ops gain a gather-rows path for short rows (`htp/gather-rows.{c,h}`, new; `cpy-ops` / `concat-ops` dispatch to it), the backend's own `htp/CMakeLists.txt` lists the file, and `scripts/snapdragon/build.py` grows options -- a backend this project does not build (no `hexagon` row in `natives.csv`; the Android jars carry the CPU modules and OpenCL only), so nothing in the range is compiled here. Root `CMakeLists.txt`, `tools/server/`, `common/`, `src/` and every patch-target file are outside the range. **No project source change.** | | b11573–b11574 | patches + upstream verification | **All twelve patches apply unchanged at the target** (`check-patches.py`: 12 patches, 99 hunks, 0 problems; the real applier `apply-llama-patches.cmake` on a fresh `FetchContent` checkout of b11574 -- `rm -rf build`, so the fetch, the twelve patches and the stamp (`head 8bb3cf0d2`, 58 modified files in the source tree) went through the production path -- then the static test build, `ctest` 607/607, and the model-free native Java tests over it: `NativeLibraryLoadSmokeTest` (the `LlamaCppVersion` pin against the linked build's `b11574-`), `LlamaLoggerTest`, `RpcServerTest`, 19/19). **The modular build** (`-DJLLAMA_BACKEND_DL=ON`, the shape of every Linux / Windows / Android library jar) **is where #30014's hidden visibility (b11570 row) could have bitten, and did not**: `libggml-base.so` exports 689 functions and `libggml.so` 16 now, every one `GGML_API` / `GGML_BACKEND_API`, among them the five the load path needs (`ggml_backend_load_all_from_path`, `ggml_backend_load`, `ggml_backend_reg_get_proc_address`, `ggml_backend_dev_count`, `ggml_log_set`); the CPU modules and `ggml-rpc` are `MODULE` targets outside the preset and export their entry points as before (390 in `libggml-cpu-haswell.so`, 77 in `libggml-rpc.so`); `JNI_OnLoad` registered RPC and CPU from the extraction directory (`2 backend(s) registered ... [RPC, CPU]`, `cascadelake` chosen on the sandbox's Xeon), and `RpcServerTest`, `LlamaLoggerTest`, `BackendLoadTest` and `NativeLibraryLoadSmokeTest` ran 27/27 over the modular library -- the RPC server resolved through `ggml_backend_reg_get_proc_address` included. Drop-checks against pristine b11574 are those of the b11573 row, the range touching no patch-target file: `split_sum` unguarded (`0012`), no `kolibri` (`0016`), no `common_log` hook (`0014`), no `stop_server` (`0015`), uncast `_mm_prefetch` (`0017`), `free(buffer->context)` (`0018`), the `GetCommandLineW` override and no `common_params_parse_main` (`0001`), the unconditional load-progress assignment (`0002`), no `LLAMA_SERVER_WORKER_CMD` (`0008`), no `get_slot_prompt_similarity` (`0003`). `check-natives.py`: 23 natives jars, 0 disagreements. Not run locally: the model-backed Java suite (no GGUF in the sandbox) -- CI's test jobs cover it. | | b11574–b11612 | 38 commits, 287 files, ~27,300/17,750 lines, 6.4 MB -- bumped in one step rather than in the runbook's < 100 KiB chunks because the bulk is data: `docs/ops/zDNN.csv` (31,433 lines), `src/unicode-data.cpp` (+3,985, #30390 opt-in NFC normalization for Unigram tokenizers), the consolidated context-state tests (#30023: `tests/test-save-load-state.cpp`, `test-recurrent-state-rollback.cpp` and `test-state-restore-fragmented.cpp` become `tests/test-llama-context.cpp` + `tests/llama-context/`), kleidiai (#29219, SME2 Q4_K/Q6_K, `KLEIDIAI_COMMIT_TAG` v1.24.0 → v1.31.0 -- ARM only, not built here) and the Vulkan/CUDA/Hexagon/zDNN backends; `tools/ui` unchanged. **Priority list:** `common/chat.h` (#30389 "chat: replace output string with parts"): `common_chat_params::prompt` and `common_chat_session::prompt()` are a `common_chat_prompt` -- a vector of `{text, is_input}` parts, `str()` renders it, `append()` / `replace_last()` skip user-input parts -- and `common_chat_template_direct_apply()` returns one; `server-common.cpp` reads `out_session.prompt().str()`. `jllama.cpp` threads the session through but never reads the prompt, so **no project source change**. `common/log.{h,cpp}` (#30023, of all things): `common_log_set_stdo()` + a `stdo` member, console output gated on it -- context of `0014`, see the patches row. `include/llama.h`: a comment on `embeddings` only; behind it #30445 ("graph: skip the LM head for pooled embeddings"): a pooled context (`pooling_type != NONE`) no longer computes logits when `embeddings` is on, `llama_set_embeddings()` toggles the graph shape (`sched_need_reserve`), and `has_logits = !cparams.embeddings || graph_has_logits` -- the embedding path only, which `LlamaEmbeddingsTest` and the langchain4j integration job exercise in CI. `tools/server/`: #30243 (`server_decision_escape_special_tokens`, `<|name|>` written with U+00A6 instead of `std::regex`) -- the decision path, no schema or response-key change (`server-schema.cpp` field set and `server-task.cpp` emit keys identical at both tags, the two mechanical checks). `src/llama-model.cpp` (#30405 "do not map foreign ranges into a backend buffer": `get_mapping_ranges()` replaces `get_mapping_range()`, `ctx_buf_maps` becomes `ctx_bufs`; #30445's `build_logits()`; `llama_model_set_attn_rot`): outside `0012`'s and `0016`'s hunks, both apply unchanged. `ggml/src/ggml-backend-reg.cpp` (#28367): `ggml_backend_load_best()` drops the pre-checks on each search path and reports an enumeration error once after the loop -- `JNI_OnLoad` calls `ggml_backend_load_all_from_path()`, unaffected. `ggml/src/ggml-cpu/arch/x86/quants.c` (+64, #30447 AVX2 Q2_0 dot product): away from `0017`'s four `_mm_prefetch` lines. Toolchain pins in upstream's `release.yml` unchanged: OpenVINO `2026.4.1` / `2026.4.1.22982.07f9c262b05`, `GGML_CUDA_CCCL_VERSION=v3.4.3`, ROCm `10.0.0`; BoringSSL `0.20260929.0`; `vendor/` untouched (cpp-httplib 0.61.0, #30494, is after b11612). Root `CMakeLists.txt` unchanged, `GGML_LLAMAFILE_DEFAULT` / `GGML_CUDA_GRAPHS_DEFAULT` still the only ggml defaults it sets. **No project source change.** | -| b11574–b11612 | patches + upstream verification | **Three of the thirteen patches needed a refresh, none was dropped.** `0001`: `tests/test-state-restore-fragmented.cpp` is gone (#30023 consolidated the context-state tests into `tests/test-llama-context.cpp`), so its hunk was dropped (36 → 35 `diff --git` headers); the consolidated test has the `--models` / `filtered_argv` `main()` and by the patch's own rule keeps `common_params_parse`. `0014`: #30023 also added `common_log_set_stdo()` and a `stdo` member exactly where the sink's members and `print_entry()`'s console call sit -- three hunks regenerated, the sink keeps precedence (`if (callback) ... else if (stdo) e.print()`), pinned by a seventh case in `test_common_log_callback.cpp`. `0006` failed only in the cascade behind `0001` (its context is the post-`0001` tree) and applies unchanged once `0001` does. The other ten apply unchanged. `check-patches.py`: 13 patches, 101 hunks, 0 problems; the real applier on a fresh `FetchContent` checkout of b11612 (stamp head `10da8c463`, 57 modified files plus the stamp), then the static test build, `ctest` **608/608**, and the model-free native Java tests over the library (`NativeLibraryLoadSmokeTest` with the `LlamaCppVersion` pin against the linked `b11612-`, `LlamaLoggerTest`, `RpcServerTest`, `BackendLoadTest`: 27/27). **Drop-checks against pristine b11612, and the one that will fire next:** `src/llama-model.cpp:1648` is still the bare `splits[i] /= split_sum` (`0012`) -- **but upstream merged the fix the same day, 15 commits past b11612**: [#29677](https://github.com/ggml-org/llama.cpp/pull/29677) / `df18bee11` ("model: split layers evenly when no device reports free memory"), the identical even-split fallback and no test. The first bump past it drops `0012`; the plan for its guard `test_model_split.cpp`, which links the two functions only the patch provides, is in `TODO.md` ("Drop patch `0012` at the first bump past b11612"). The rest unchanged: no `kolibri` under `src/` (`0016`); `common/log.h` has a `set_stdo` but no hook, only `common_log_default_callback` (`0014`); no `stop_server`, the connect failure still aborts at `ggml-rpc.cpp:645` (`0015`); the four `_mm_prefetch` calls at `quants.c:846-865` still uncast (`0017`); `amx.cpp:46` still `free(buffer->context)` (`0018`); the `__ARM_NEON` branch of `simd-mappings.h` still selects the software conversion (`0019`); `arg.cpp:1288` still carries the `GetCommandLineW` override and no `common_params_parse_main` (`0001`); `server-context.cpp:1212` still assigns the load-progress callback unconditionally (`0002`); no `LLAMA_SERVER_WORKER_CMD` (`0008`), no `get_slot_prompt_similarity` (`0003`). `check-natives.py`: 23 natives jars, 0 disagreements; spotless clean. Not run locally: the model-backed Java suite (no GGUF in the sandbox) and the modular build -- CI's test and smoke jobs cover both. | +| b11574–b11612 | patches + upstream verification | **Three of the thirteen patches needed a refresh, none was dropped.** `0001`: `tests/test-state-restore-fragmented.cpp` is gone (#30023 consolidated the context-state tests into `tests/test-llama-context.cpp`), so its hunk was dropped (36 → 35 `diff --git` headers); the consolidated test has the `--models` / `filtered_argv` `main()` and by the patch's own rule keeps `common_params_parse`. `0014`: #30023 also added `common_log_set_stdo()` and a `stdo` member exactly where the sink's members and `print_entry()`'s console call sit -- three hunks regenerated, the sink keeps precedence (`if (callback) ... else if (stdo) e.print()`), pinned by a seventh case in `test_common_log_callback.cpp`. `0006` failed only in the cascade behind `0001` (its context is the post-`0001` tree) and applies unchanged once `0001` does. The other ten apply unchanged. `check-patches.py`: 13 patches, 101 hunks, 0 problems; the real applier on a fresh `FetchContent` checkout of b11612 (stamp head `10da8c463`, 57 modified files plus the stamp), then the static test build, `ctest` **608/608**, and the model-free native Java tests over the library (`NativeLibraryLoadSmokeTest` with the `LlamaCppVersion` pin against the linked `b11612-`, `LlamaLoggerTest`, `RpcServerTest`, `BackendLoadTest`: 27/27). **Drop-checks against pristine b11612, and the one that will fire next:** `src/llama-model.cpp:1648` is still the bare `splits[i] /= split_sum` (`0012`) -- **but upstream merged the fix the same day, 15 commits past b11612**: [#29677](https://github.com/ggml-org/llama.cpp/pull/29677) / `df18bee11` ("model: split layers evenly when no device reports free memory"), the identical even-split fallback and no test. The first bump past it drops `0012`; the plan for its guard `test_model_split.cpp`, which links the two functions only the patch provides, is in `TODO.md` ("Drop patch `0012` at the first bump that carries #29677"). The rest unchanged: no `kolibri` under `src/` (`0016`); `common/log.h` has a `set_stdo` but no hook, only `common_log_default_callback` (`0014`); no `stop_server`, the connect failure still aborts at `ggml-rpc.cpp:645` (`0015`); the four `_mm_prefetch` calls at `quants.c:846-865` still uncast (`0017`); `amx.cpp:46` still `free(buffer->context)` (`0018`); the `__ARM_NEON` branch of `simd-mappings.h` still selects the software conversion (`0019`); `arg.cpp:1288` still carries the `GetCommandLineW` override and no `common_params_parse_main` (`0001`); `server-context.cpp:1212` still assigns the load-progress callback unconditionally (`0002`); no `LLAMA_SERVER_WORKER_CMD` (`0008`), no `get_slot_prompt_similarity` (`0003`). `check-natives.py`: 23 natives jars, 0 disagreements; spotless clean. Not run locally: the model-backed Java suite (no GGUF in the sandbox) and the modular build -- CI's test and smoke jobs cover both. | +| b11612–b11617 | 5 commits, 5 files, ~99/85 lines (9 KiB; no `tools/ui`, no `.github`, no `vendor`). Four are Vulkan (#29069 bf16 activations for mulmat, #30431 the shader count for Intel Arc Pro B65/B70, #29835 a workaround for a device reporting free memory above total on Intel Windows, #30495 a compile fix) -- the module jar `vulkan-*` rebuilds them, nothing here reads them. One is `src/models/` (#30480 "llama : build the k-pool selection in the causal mask layout"): `glm5-next.cpp` and `qwen4exp.cpp` hand the DSA/QSA selection to `build_attn_mha` in the mask's own 4-D layout instead of reshaping it, with `ggml_are_same_shape` asserts -- graph-internal to two architectures, outside every patch-target file. Nothing on the priority list changed; `src/llama-model.cpp` is untouched (`split_sum` still unguarded at `:1648`, `0012` still carried -- `df18bee11` / #29677 is in no tag up to b11618). Toolchain pins unchanged. **No project source change.** | +| b11612–b11617 | patches + upstream verification | **All thirteen patches apply unchanged at b11617** (`git apply` in order on a pristine b11617 worktree, 57 modified files; `check-patches.py`: 13 patches, 101 hunks, 0 problems; the real applier on a fresh `FetchContent` checkout -- `rm -rf build` -- with stamp head `14310190c`), then the static test build, `ctest` **608/608**, and the model-free native Java tests over the library (`NativeLibraryLoadSmokeTest` with the `LlamaCppVersion` pin against the linked `b11617-`, `LlamaLoggerTest`, `RpcServerTest`, `BackendLoadTest`: 27/27). Drop-checks against pristine b11617 are those of the b11612 row, the range touching no patch-target file: `split_sum` unguarded at `src/llama-model.cpp:1648` (`0012`; the upstream fix `df18bee11` / #29677 is still in no tag -- newest b11618 -- so the drop rule in `TODO.md` keys on the fixing commit now, not on "past b11612"), no `kolibri` (`0016`), no `common_log` hook (`0014`), no `stop_server` (`0015`), uncast `_mm_prefetch` (`0017`), `free(buffer->context)` (`0018`), the software fp16 conversion on `__ARM_NEON` (`0019`), the `GetCommandLineW` override and no `common_params_parse_main` (`0001`), the unconditional load-progress assignment (`0002`), no `LLAMA_SERVER_WORKER_CMD` (`0008`), no `get_slot_prompt_similarity` (`0003`). `check-natives.py`: 23 natives jars, 0 disagreements. Not run locally: the model-backed Java suite and the modular build -- CI covers both. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index 454a5dab..da22a13a 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -337,7 +337,7 @@ endif() FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b11612 + GIT_TAG b11617 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/patches/0012-model-guard-zero-split-sum-and-name-the-device-index.patch b/llama/patches/0012-model-guard-zero-split-sum-and-name-the-device-index.patch index ab20e866..df9ff08f 100644 --- a/llama/patches/0012-model-guard-zero-split-sum-and-name-the-device-index.patch +++ b/llama/patches/0012-model-guard-zero-split-sum-and-name-the-device-index.patch @@ -1,19 +1,19 @@ model: fall back to an even split when no device reports free memory, and name the device index on a bad lookup (the two load_tensors() blocks lifted into free functions so they can be tested). -STATUS 2026-10-14: FIXED UPSTREAM -- DROP THIS PATCH AT THE FIRST llama.cpp BUMP PAST b11612. +STATUS 2026-10-14: FIXED UPSTREAM -- DROP THIS PATCH AT THE FIRST llama.cpp BUMP TO A TAG THAT CARRIES df18bee11. ggml-org/llama.cpp#29677, commit df18bee11 ("model: split layers evenly when no device reports free memory"), https://github.com/ggml-org/llama.cpp/commit/df18bee11e96557b07aa7b12db0976f3c7ac9165, puts the same even-split fallback into load_tensors(): splits[i] = split_sum != 0.0f ? splits[i] / split_sum : float(i + 1) / n_devices(); -It is NOT in b11612 (10da8c463, 15 commits earlier on master), so the patch is still needed at that -pin and applies unchanged. Drop-check: `git grep -n split_sum src/llama-model.cpp` at the new tag -- +It is NOT in b11612 (10da8c463, 15 commits earlier on master) nor in b11617, so the patch is still +needed at those pins and applies unchanged. Drop-check: `git grep -n split_sum src/llama-model.cpp` at the new tag -- the guarded line means DROP, not refresh (the patch keeps applying next to it, so the applier will not notice). Upstream ships no test. The runnable guard src/test/cpp/test_model_split.cpp links the two functions this patch adds and cannot outlive it as it is; what replaces it is decided in -TODO.md, "Drop patch 0012 at the first bump past b11612". +TODO.md, "Drop patch 0012 at the first bump that carries #29677". diff --git a/src/llama-model.cpp b/src/llama-model.cpp index 6872589fa..a56e4bb04 100644 diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index 7ed99d7a..4a816003 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -9,13 +9,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b11612"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b11617"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b11612-"} — call + * plus the resolved upstream commit, e.g. {@code "b11617-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -23,14 +23,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b11612"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b11617"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b11612"; + public static final String LLAMA_CPP_VERSION = "b11617"; // Constants holder — not instantiable. private LlamaCppVersion() {} diff --git a/llama/src/test/cpp/test_model_split.cpp b/llama/src/test/cpp/test_model_split.cpp index fcfddf0e..82194c51 100644 --- a/llama/src/test/cpp/test_model_split.cpp +++ b/llama/src/test/cpp/test_model_split.cpp @@ -16,10 +16,11 @@ // // Upstream fixed the defect itself on 2026-10-14 (ggml-org/llama.cpp#29677, commit df18bee11, // https://github.com/ggml-org/llama.cpp/commit/df18bee11e96557b07aa7b12db0976f3c7ac9165): the same -// even-split fallback, inline in load_tensors(), and no test. The commit is not in b11612, the pin at -// the time of writing, so the patch and this guard still stand there. At the first bump past it -// patches/0012 is dropped, and with it the two functions this file links -- see TODO.md, "Drop patch -// 0012 at the first bump past b11612", for what replaces this guard. +// even-split fallback, inline in load_tensors(), and no test. The commit is in no tag up to b11618 +// (b11617 is the pin at the time of writing), so the patch and this guard still stand. At the first +// bump to a tag that carries it, patches/0012 is dropped, and with it the two functions this file +// links -- see TODO.md, "Drop patch 0012 at the first bump that carries #29677", for what replaces +// this guard. #include "llama-model.h" From b5db648d32538631071026475172b6a1aee67859 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 14 Oct 2026 14:02:07 +0000 Subject: [PATCH 2/3] Pin llama.cpp by number, with an optional commit for a number upstream has not tagged yet The pin is set(JLLAMA_GIT_TAG b) now, and everything that names the version keeps reading the number: jllama-build.txt, LlamaCppVersion, the README badge, the WebUI job, llama-next-version.sh. A number upstream has not tagged yet -- the tags trail master, and a fix worth bumping to can sit in between -- is pinned the same way plus set(JLLAMA_GIT_COMMIT ""), the master commit the number names. Only the two fetches read the hash (FetchContent and the build-webui checkout); the configure then asserts that the fetched commit's `git rev-list --count` is the number, which is upstream's own tag-numbering rule and also how llama.cpp's build-info derives b-, so the Java pin check holds against a virtual number unchanged. llama-next-version.sh resolves a number to its hash (--resolve b), accepts a virtual current pin, and says when JLLAMA_GIT_COMMIT is stale because the tag has been published. Measured on three configures: fetched by tag (count 11617 = b11617), fetched by b11617's commit with JLLAMA_GIT_COMMIT set (same), and fetched by b11618's commit under JLLAMA_GIT_TAG b11617, which the pin check refuses naming the number the hash really is. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_017kVXQfPe1ACLAuk29Y4d2H --- .github/scripts/llama-next-version.sh | 84 +++++++++++++++++++++----- .github/workflows/publish.yml | 13 ++-- CLAUDE.md | 32 ++++++---- docs/upgrade/llama-cpp-version-bump.md | 39 ++++++++++-- llama/CMakeLists.txt | 71 +++++++++++++++++++--- 5 files changed, 192 insertions(+), 47 deletions(-) diff --git a/.github/scripts/llama-next-version.sh b/.github/scripts/llama-next-version.sh index d2d48acc..ef680bbe 100755 --- a/.github/scripts/llama-next-version.sh +++ b/.github/scripts/llama-next-version.sh @@ -8,7 +8,12 @@ # The runbook this supports is docs/upgrade/llama-cpp-version-bump.md. Strategy: # * TARGET = the topmost RELEASE on the GitHub releases page (read from the release atom feed), # or an explicit "b" passed as $1. -# * CURRENT = the pinned tag in llama/CMakeLists.txt (GIT_TAG b). +# * CURRENT = the pinned number in llama/CMakeLists.txt (set(JLLAMA_GIT_TAG b)). It may be a +# "virtual" number -- one upstream has not tagged yet, pinned through JLLAMA_GIT_COMMIT -- +# in which case it is resolved to the master commit whose `git rev-list --count` is the +# number, the way upstream's release.yml numbers its tags. +# * `--resolve b` prints that commit for any number (the tag's commit when the tag exists) and +# exits: the hash to put into JLLAMA_GIT_COMMIT. # * If `git diff CURRENT..TARGET` is smaller than the threshold (default 100 KiB), bump straight # to TARGET. Otherwise CHUNK: pick the largest intermediate b tag whose diff from CURRENT # is still under the threshold, so each bump stays a small, reviewable patch. Re-run after each @@ -36,9 +41,10 @@ CACHE="${LLAMA_BUMP_CACHE:-$HOME/.cache/jllama-llamacpp-mirror}" ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" CMAKELISTS="$ROOT/llama/CMakeLists.txt" -# --- current pinned tag number, e.g. "GIT_TAG b9866" -> 9866 ----------------------------------- -cur="$(grep -oE 'GIT_TAG[[:space:]]+b[0-9]+' "$CMAKELISTS" | grep -oE '[0-9]+' | head -1 || true)" -[ -n "$cur" ] || { echo "ERROR: could not read 'GIT_TAG b' from $CMAKELISTS" >&2; exit 1; } +# --- current pinned number, e.g. "set(JLLAMA_GIT_TAG b9866)" -> 9866, and the optional commit ------ +cur="$(grep -oE '^set\(JLLAMA_GIT_TAG[[:space:]]+b[0-9]+\)' "$CMAKELISTS" | grep -oE '[0-9]+' | head -1 || true)" +[ -n "$cur" ] || { echo "ERROR: could not read 'set(JLLAMA_GIT_TAG b)' from $CMAKELISTS" >&2; exit 1; } +pinned_commit="$(grep -oE '^set\(JLLAMA_GIT_COMMIT[[:space:]]+"[0-9a-f]{40}"\)' "$CMAKELISTS" | grep -oE '[0-9a-f]{40}' | head -1 || true)" # --- cached blobless mirror of llama.cpp (clone once, then fetch tags) -------------------------- if [ -d "$CACHE/.git" ]; then @@ -48,6 +54,40 @@ else git clone --filter=blob:none --no-checkout --quiet "$GIT_URL" "$CACHE" fi +# --- a number -> the ref that carries it: the tag when upstream published it, else the master ----- +# commit whose `git rev-list --count` is the number (upstream's own numbering rule). A virtual number +# is recent by nature, so the walk down first-parent master is short; it stops at the first commit +# whose count is <= the number and errors when that count is not the number (a number no master +# commit carries, or one further back than the walk looks). +ref_of_number() { + local n="$1" c count i=0 + if git -C "$CACHE" rev-parse -q --verify "b${n}^{commit}" >/dev/null 2>&1; then + echo "b$n"; return 0 + fi + git -C "$CACHE" fetch --quiet origin master 2>/dev/null || true + while read -r c; do + count="$(git -C "$CACHE" rev-list --count "$c")" + if [ "$count" -eq "$n" ]; then echo "$c"; return 0; fi + if [ "$count" -lt "$n" ]; then break; fi + i=$((i + 1)); [ "$i" -lt 2000 ] || break + done < <(git -C "$CACHE" rev-list --first-parent origin/master) + echo "ERROR: b$n is not a tag in the mirror and no recent master commit has commit count $n" >&2 + return 1 +} + +if [ "${1:-}" = "--resolve" ]; then + n="$(printf '%s' "${2:-}" | grep -oE '[0-9]+' | head -1)" + [ -n "$n" ] || { echo "usage: $0 --resolve b" >&2; exit 1; } + ref="$(ref_of_number "$n")" || exit 3 + sha="$(git -C "$CACHE" rev-parse "${ref}^{commit}")" + if [ "$ref" = "b$n" ]; then + echo "b$n is published: $sha (JLLAMA_GIT_COMMIT can stay empty)" + else + echo "b$n is not published yet; it will be $sha -- set(JLLAMA_GIT_COMMIT \"$sha\")" + fi + exit 0 +fi + # --- target: explicit "$1" (b) or the latest RELEASE from the atom feed ------------------- if [ "${1:-}" != "" ]; then target="$(printf '%s' "$1" | grep -oE '[0-9]+' | head -1)" @@ -59,21 +99,33 @@ else [ -n "$target" ] || { echo "ERROR: parsed no release tags from the feed." >&2; exit 3; } fi -git -C "$CACHE" rev-parse -q --verify "b${cur}^{commit}" >/dev/null 2>&1 || { echo "ERROR: b$cur is not a tag in the mirror" >&2; exit 3; } +curref="$(ref_of_number "$cur")" || exit 3 git -C "$CACHE" rev-parse -q --verify "b${target}^{commit}" >/dev/null 2>&1 || { echo "ERROR: b$target is not a tag in the mirror" >&2; exit 3; } +if [ -n "$pinned_commit" ]; then + # JLLAMA_GIT_COMMIT is set: it must be the commit the number names, and it is stale once the tag exists + pinned_count="$(git -C "$CACHE" rev-list --count "$pinned_commit" 2>/dev/null || true)" + if [ "$pinned_count" != "$cur" ]; then + echo "ERROR: JLLAMA_GIT_COMMIT $pinned_commit has commit count ${pinned_count:-?}, not $cur -- it does not resolve b$cur (run: $0 --resolve b$cur)" >&2; exit 3 + fi + if [ "$curref" = "b$cur" ]; then + echo "NOTE: upstream has published b$cur ($(git -C "$CACHE" rev-parse "b$cur^{commit}")): clear JLLAMA_GIT_COMMIT in $CMAKELISTS, the tag carries the pin now." >&2 + fi +elif [ "$curref" != "b$cur" ]; then + echo "NOTE: b$cur is not a published tag and JLLAMA_GIT_COMMIT is empty -- the native fetch will fail; set(JLLAMA_GIT_COMMIT \"$curref\")" >&2 +fi -# diff byte size between two tag numbers, honoring the WebUI-exclusion toggle +# diff byte size between the current ref and a tag number, honoring the WebUI-exclusion toggle diffsize() { if [ "$EXCLUDE_WEBUI" = "1" ]; then - git -C "$CACHE" diff "b$1" "b$2" -- . ':(exclude)tools/ui' 2>/dev/null | wc -c + git -C "$CACHE" diff "$1" "b$2" -- . ':(exclude)tools/ui' 2>/dev/null | wc -c else - git -C "$CACHE" diff "b$1" "b$2" 2>/dev/null | wc -c + git -C "$CACHE" diff "$1" "b$2" 2>/dev/null | wc -c fi } scope="full diff" [ "$EXCLUDE_WEBUI" = "1" ] && scope="diff excluding tools/ui" -echo "current pin : b$cur" +echo "current pin : b$cur$( [ "$curref" = "b$cur" ] || echo " (virtual: not tagged upstream yet, resolved to $curref)")" echo "latest release : b$target" echo "threshold : ${THRESHOLD_KB} KiB per step (${scope})" @@ -83,7 +135,7 @@ if [ "$cur" -ge "$target" ]; then fi # --- choose next step: TARGET if it fits, else the largest intermediate tag under the threshold - -if [ "$(diffsize "$cur" "$target")" -lt "$THRESHOLD" ]; then +if [ "$(diffsize "$curref" "$target")" -lt "$THRESHOLD" ]; then next="$target" else # existing b-tags strictly after cur, up to and including target, ascending @@ -95,7 +147,7 @@ else lo=0; hi=$(( ${#cands[@]} - 1 )); best="" while [ "$lo" -le "$hi" ]; do mid=$(( (lo + hi) / 2 )); T="${cands[$mid]}" - if [ "$(diffsize "$cur" "$T")" -lt "$THRESHOLD" ]; then best="$T"; lo=$(( mid + 1 )); else hi=$(( mid - 1 )); fi + if [ "$(diffsize "$curref" "$T")" -lt "$THRESHOLD" ]; then best="$T"; lo=$(( mid + 1 )); else hi=$(( mid - 1 )); fi done if [ -n "$best" ]; then next="$best" @@ -105,9 +157,9 @@ else fi fi -full=$(git -C "$CACHE" diff "b$cur" "b$next" | wc -c) -noui=$(git -C "$CACHE" diff "b$cur" "b$next" -- . ':(exclude)tools/ui' | wc -c) -commits=$(git -C "$CACHE" rev-list --count "b$cur".."b$next") +full=$(git -C "$CACHE" diff "$curref" "b$next" | wc -c) +noui=$(git -C "$CACHE" diff "$curref" "b$next" -- . ':(exclude)tools/ui' | wc -c) +commits=$(git -C "$CACHE" rev-list --count "$curref".."b$next") echo echo "next step : b$cur -> b$next" echo " diff size : $((full / 1024)) KiB full / $((noui / 1024)) KiB excluding tools/ui (auto-followed WebUI)" @@ -117,7 +169,7 @@ if [ "$next" -eq "$target" ]; then else echo " progress : intermediate chunk — re-run this script after the bump for the next one" fi -echo " review diff : https://github.com/${REPO}/compare/b$cur...b$next" -echo " raw .patch : https://github.com/${REPO}/compare/b$cur...b$next.patch" +echo " review diff : https://github.com/${REPO}/compare/$curref...b$next" +echo " raw .patch : https://github.com/${REPO}/compare/$curref...b$next.patch" echo echo "Apply this bump per docs/upgrade/llama-cpp-version-bump.md (b$cur -> b$next)." diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 2b5d87ff..aa8aa4ca 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -538,18 +538,23 @@ jobs: id: tag shell: bash run: | - TAG=$(grep -oE 'GIT_TAG[[:space:]]+b[0-9]+' llama/CMakeLists.txt | grep -oE 'b[0-9]+' | head -1) + # The pin is JLLAMA_GIT_TAG (b); JLLAMA_GIT_COMMIT, when set, is the commit that number + # names while upstream has not published the tag -- the checkout takes it, the UI build keeps + # the number (the same rule llama/CMakeLists.txt applies to the native fetch). + TAG=$(grep -oE '^set\(JLLAMA_GIT_TAG[[:space:]]+b[0-9]+\)' llama/CMakeLists.txt | grep -oE 'b[0-9]+' | head -1) if [ -z "$TAG" ]; then - echo "could not resolve llama.cpp GIT_TAG (b) from CMakeLists.txt" >&2 + echo "could not resolve llama.cpp JLLAMA_GIT_TAG (b) from CMakeLists.txt" >&2 exit 1 fi + COMMIT=$(grep -oE '^set\(JLLAMA_GIT_COMMIT[[:space:]]+"[0-9a-f]{40}"\)' llama/CMakeLists.txt | grep -oE '[0-9a-f]{40}' | head -1 || true) echo "tag=$TAG" >> "$GITHUB_OUTPUT" - echo "Pinned llama.cpp WebUI tag: $TAG" + echo "ref=${COMMIT:-$TAG}" >> "$GITHUB_OUTPUT" + echo "Pinned llama.cpp WebUI tag: $TAG${COMMIT:+ (fetched by commit $COMMIT: the tag is not published yet)}" - name: Checkout llama.cpp tools/ui at the pinned tag uses: actions/checkout@v7 with: repository: ggml-org/llama.cpp - ref: ${{ steps.tag.outputs.tag }} + ref: ${{ steps.tag.outputs.ref }} path: llamacpp-ui # tools/ui holds the npm project AND the ui.cpp.in / ui.h.in templates; # scripts/ holds ui-assets.cmake, which consumes them. Both are needed diff --git a/CLAUDE.md b/CLAUDE.md index 02c68801..e16582d0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -161,8 +161,8 @@ can join it) and macOS (`metal-macos-aarch64`: Metal is compiled into the librar macOS static too). There the loader extracts and loads a single file (plus `jllama-extras.txt` and a `ggml-metal.metal` a build does not embed). -**Every directory carries `jllama-build.txt`** (`llama.cpp=` parsed out of this file's own -`GIT_TAG` line, `backend=`, `kind=library|module`), written with `file(GENERATE)`: the loader's build +**Every directory carries `jllama-build.txt`** (`llama.cpp=`, the `JLLAMA_GIT_TAG` number of this +file's pin, `backend=`, `kind=library|module`), written with `file(GENERATE)`: the loader's build check and the merge step's completeness check read it (see "Natives jars"). Before 5.2.0 each jar was **one** library for one level -- x86-64 the Haswell baseline (x86-64-v3, @@ -1071,7 +1071,8 @@ checked in (same policy as the native libs). Pipeline (`.github/workflows/publish.yml`): 1. **`build-webui` job** (ubuntu — the *only* job that runs `npm`): resolves the - pinned `b` tag from `llama/CMakeLists.txt`'s `GIT_TAG`, sparse-checks-out + pinned `b` tag from `llama/CMakeLists.txt`'s `JLLAMA_GIT_TAG` (checking out + `JLLAMA_GIT_COMMIT` instead when that is set, see "Upgrading/Downgrading"), sparse-checks-out `ggml-org/llama.cpp@` `tools/ui`, runs the upstream Svelte build (`npm ci && npm run build`), gzips `dist/` into `dist/_gzip/` (LLAMA_UI_GZIP parity), then runs upstream's own **`scripts/ui-assets.cmake`** (a plain `cmake -P` @@ -1099,7 +1100,7 @@ Pipeline (`.github/workflows/publish.yml`): empty-asset stub `src/main/cpp/webui_stub/ui.h` (no embedded UI) so local builds — and any job without the artifact — still build and run. -The WebUI version **auto-follows** the pinned `GIT_TAG`: a llama.cpp version bump +The WebUI version **auto-follows** the pinned `JLLAMA_GIT_TAG`: a llama.cpp version bump needs no extra step here, `build-webui` re-reads the tag and rebuilds the matching UI. **Building the WebUI locally** (optional — a plain `cmake` build uses the stub and @@ -1146,7 +1147,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b11617`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`JLLAMA_GIT_TAG b11617`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1615,21 +1616,30 @@ edit/verify/commit loop below. Use it for any non-trivial bump; the steps here a To change the llama.cpp version, update the following **four** files (and re-verify `patches/`): -1. **llama/CMakeLists.txt** — the `GIT_TAG` line for llama.cpp: `GIT_TAG b8831`. (There is - no second tag to keep in sync any more: the cosmetic `-DLLAMA_TAG=` that fed the old build-time - TTS extraction went away with the Qwen3-TTS rework — see "Qwen3-TTS via `mtmd_helper::gen_audio`".) +1. **llama/CMakeLists.txt** — the pin: `set(JLLAMA_GIT_TAG b8831)`. (There is no second tag to + keep in sync any more: the cosmetic `-DLLAMA_TAG=` that fed the old build-time TTS extraction went + away with the Qwen3-TTS rework — see "Qwen3-TTS via `mtmd_helper::gen_audio`".) **A number + upstream has not tagged yet** (the tags trail master, and a fix worth bumping to may sit in + between) is pinned the same way plus `set(JLLAMA_GIT_COMMIT "")`, the commit the + number names -- `llama-next-version.sh --resolve b` prints it. Only the two fetches + (`FetchContent`, the `build-webui` checkout) read the hash; `jllama-build.txt`, the loader, + `LlamaCppVersion`, README and this file keep the number, and the configure asserts that the + fetched commit's `git rev-list --count` is the number (upstream's own tag-numbering rule, and + also how llama.cpp's `build-info` derives `b-` -- so the Java pin check holds + against a virtual number too). Clear `JLLAMA_GIT_COMMIT` once the tag exists; the helper prints a + NOTE while it is stale. Runbook: "Pinning a number upstream has not tagged yet". 2. **README.md** — the badge and link line with the version number 3. **CLAUDE.md** — the "Current llama.cpp pinned version" line 4. **llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java** — the `LLAMA_CPP_VERSION` constant (the compile-time pin exposed to Java/Kotlin consumers, e.g. the - Android version badge). It is the *pure-Java mirror* of `GIT_TAG` and must stay equal to it — + Android version badge). It is the *pure-Java mirror* of `JLLAMA_GIT_TAG` and must stay equal to it — the native `LlamaModel.getLlamaCppBuildInfo()` getter reports the actually-linked build (`b-`), and `NativeLibraryLoadSmokeTest.nativeBuildInfoMatchesPinnedVersionConstant` **fails the build** if this constant and the linked binary drift apart. Example: To upgrade from b8808 to b8831: ```bash -# Edit llama/CMakeLists.txt: change GIT_TAG b8808 to b8831 +# Edit llama/CMakeLists.txt: change JLLAMA_GIT_TAG b8808 to b8831 # Edit README.md: change b8808 to b8831 (in both badge and link) # Edit CLAUDE.md: change b8808 to b8831 # Edit LlamaCppVersion.java: change LLAMA_CPP_VERSION "b8808" to "b8831" @@ -2412,7 +2422,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b11617`. +llama.cpp is fetched via CMake FetchContent, pinned to `JLLAMA_GIT_TAG b11617`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.18.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/docs/upgrade/llama-cpp-version-bump.md b/docs/upgrade/llama-cpp-version-bump.md index fbca0e09..b5b12d4d 100644 --- a/docs/upgrade/llama-cpp-version-bump.md +++ b/docs/upgrade/llama-cpp-version-bump.md @@ -11,8 +11,32 @@ mechanical edit steps in [`../../CLAUDE.md`](../../CLAUDE.md#upgradingdowngradin together with a repeatable **target-selection + chunking** strategy so a bump never lands an unreviewably large diff in one step. -The current pin lives in `llama/CMakeLists.txt` as `GIT_TAG b`. llama.cpp tags **every** -master commit as `b`, but only a subset get GitHub *Releases*. +The current pin lives in `llama/CMakeLists.txt` as `set(JLLAMA_GIT_TAG b)`. llama.cpp tags +**every** master commit as `b`, but only a subset get GitHub *Releases*. + +## Pinning a number upstream has not tagged yet (a virtual b-number) + +The number is the pin everywhere -- `jllama-build.txt`, `LlamaCppVersion`, the README badge, the +WebUI job -- and it is derivable: upstream's `release.yml` numbers a tag by `git rev-list --count` +of the master commit, so `b` names exactly one commit whether or not the tag has been pushed +yet (the tags trail master by hours to days). When a bump wants a commit that is already on master +but not yet tagged, keep the number and name the commit next to it: + +```cmake +set(JLLAMA_GIT_TAG b11626) +set(JLLAMA_GIT_COMMIT "df18bee11e96557b07aa7b12db0976f3c7ac9165") # until upstream tags b11626 +``` + +`.github/scripts/llama-next-version.sh --resolve b` prints that hash (from the mirror clone, +walking master for the commit whose count is the number). Only the two fetches read the commit -- +`FetchContent` in `llama/CMakeLists.txt` and the `build-webui` checkout -- and the configure then +**asserts** that the fetched commit's count is the number (`llama.cpp pin check` in the CMake +output), so a wrong hash cannot ship as `b`; llama.cpp's own `build-info` derives its number +the same way, which is why `NativeLibraryLoadSmokeTest` passes against a virtual number too. Every +other consumer keeps the number. Once upstream publishes the tag, clear `JLLAMA_GIT_COMMIT` +(`llama-next-version.sh` prints a NOTE when it is stale); nothing else changes. Two limits: the +commit must be on master (a side branch's count is not a tag number), and the README's release link +points at a page that does not exist until the tag does. --- @@ -123,14 +147,17 @@ Once you have the `b -> b` step, apply it exactly as Concretely: 1. **Edit the pin — four files:** - - `llama/CMakeLists.txt` — the `GIT_TAG b` line. (It is the only `b` tag in this file — the other two `GIT_TAG` lines pin nlohmann/json `v3.12.0` and GoogleTest `v1.18.0` and must NOT move with a llama.cpp bump. The + - `llama/CMakeLists.txt` — the `set(JLLAMA_GIT_TAG b)` line (and `JLLAMA_GIT_COMMIT`, see + "Pinning a number upstream has not tagged yet": set it when the number is not tagged yet, clear + it when it is). It is the only `b` in this file — the two `GIT_TAG` lines of nlohmann/json + `v3.12.0` and GoogleTest `v1.18.0` must NOT move with a llama.cpp bump. The `-DLLAMA_TAG=b` that once fed the build-time TTS extraction was removed with the - Qwen3-TTS rework, and the WebUI auto-follows `GIT_TAG` in CI.) + Qwen3-TTS rework, and the WebUI auto-follows `JLLAMA_GIT_TAG` in CI.) - `README.md` — the llama.cpp badge and link (the tag appears three times, all on one line: the badge alt text, the badge URL and the release link). - `CLAUDE.md` — the "Current llama.cpp pinned version" line (and any build-example `b`). - `llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java` — the `LLAMA_CPP_VERSION` - constant (the pure-Java pin consumers read for a version badge/log line). It mirrors `GIT_TAG`; + constant (the pure-Java pin consumers read for a version badge/log line). It mirrors `JLLAMA_GIT_TAG`; if you forget it, `NativeLibraryLoadSmokeTest.nativeBuildInfoMatchesPinnedVersionConstant` fails the build (it cross-checks the constant against `LlamaModel.getLlamaCppBuildInfo()`, which reads llama.cpp's own linked-in `build-info`). @@ -151,7 +178,7 @@ Concretely: else: `apply-llama-patches.cmake` `file(GLOB)`s the directory and applies them in filename order, so an enumeration can only go stale (it did, one commit after being written). Use a **fresh** build dir: the applier's stamp file pins the patch set to the *checked-out llama.cpp commit*, so - after a `GIT_TAG` change an existing build dir is exactly the case it refuses to guess at — it + after a pin change an existing build dir is exactly the case it refuses to guess at — it aborts and tells you to configure fresh, which is what actually re-runs the patches against the new source: ```bash diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index da22a13a..f8a900cf 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -334,10 +334,39 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) if(JLLAMA_BACKEND_DL) set(JLLAMA_LLAMA_SOURCE_SUBDIR SOURCE_SUBDIR jllama-added-below) endif() +# ---- The llama.cpp pin ------------------------------------------------------------------------ +# JLLAMA_GIT_TAG is THE pin, a b release number, and everything that names the version reads +# it: jllama-build.txt (the loader's same-build check), the build-webui job, llama-next-version.sh, +# and by hand README.md, CLAUDE.md and LlamaCppVersion.java. Upstream numbers its tags by +# `git rev-list --count` of the master commit, so a number names exactly one commit whether or not +# upstream has pushed the tag yet. That is what JLLAMA_GIT_COMMIT is for: while the tag is not +# published, set it to that commit's full hash (a "virtual" b-number) -- the fetch below takes the +# hash, every other consumer keeps the number, and the pin check after FetchContent_MakeAvailable +# asserts that the fetched commit's count IS the number. Clear it once the tag exists; +# `llama-next-version.sh` says when, and `llama-next-version.sh --resolve b` prints the hash. +set(JLLAMA_GIT_TAG b11617) +set(JLLAMA_GIT_COMMIT "") +if(NOT JLLAMA_GIT_TAG MATCHES "^b[0-9]+$") + message(FATAL_ERROR "JLLAMA_GIT_TAG must be a llama.cpp release number b, got '${JLLAMA_GIT_TAG}'") +endif() +if(JLLAMA_GIT_COMMIT AND NOT JLLAMA_GIT_COMMIT MATCHES "^[0-9a-f]+$") + message(FATAL_ERROR "JLLAMA_GIT_COMMIT must be empty or a full 40-hex commit hash, got '${JLLAMA_GIT_COMMIT}'") +endif() +if(JLLAMA_GIT_COMMIT) + string(LENGTH "${JLLAMA_GIT_COMMIT}" _jllama_commit_len) + if(NOT _jllama_commit_len EQUAL 40) + message(FATAL_ERROR "JLLAMA_GIT_COMMIT must be the full 40-hex commit hash, got '${JLLAMA_GIT_COMMIT}' (${_jllama_commit_len} chars)") + endif() + set(_jllama_fetch_ref "${JLLAMA_GIT_COMMIT}") + message(STATUS "llama.cpp ${JLLAMA_GIT_TAG} is fetched by commit ${JLLAMA_GIT_COMMIT} (JLLAMA_GIT_COMMIT is set: upstream has not published the tag)") +else() + set(_jllama_fetch_ref "${JLLAMA_GIT_TAG}") +endif() + FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b11617 + GIT_TAG ${_jllama_fetch_ref} PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= @@ -346,6 +375,33 @@ FetchContent_Declare( ) FetchContent_MakeAvailable(llama.cpp) +# llama.cpp pin check: the fetched commit's count must be the number JLLAMA_GIT_TAG names. Fetched by +# tag, that is upstream's own invariant (release.yml numbers the tag by that count); fetched by +# JLLAMA_GIT_COMMIT, it is the proof that the hash resolves the number -- a wrong hash would otherwise +# ship as b with another commit's code, and the Java pin check (NativeLibraryLoadSmokeTest +# against build-info, which derives its number the same way) would see it only after the natives +# were built. Skipped, with a message, where it cannot be answered: no git, a source tree that is not +# a git checkout (-DFETCHCONTENT_SOURCE_DIR_LLAMA.CPP), or a shallow clone (count is meaningless). +find_package(Git QUIET) +if(GIT_EXECUTABLE AND EXISTS "${llama.cpp_SOURCE_DIR}/.git" AND NOT EXISTS "${llama.cpp_SOURCE_DIR}/.git/shallow") + string(REGEX REPLACE "^b" "" _jllama_tag_number "${JLLAMA_GIT_TAG}") + execute_process( + COMMAND "${GIT_EXECUTABLE}" rev-list --count HEAD + WORKING_DIRECTORY "${llama.cpp_SOURCE_DIR}" + OUTPUT_VARIABLE _jllama_head_count + OUTPUT_STRIP_TRAILING_WHITESPACE + RESULT_VARIABLE _jllama_count_res) + if(NOT _jllama_count_res EQUAL 0) + message(WARNING "llama.cpp pin check skipped: `git rev-list --count HEAD` failed in ${llama.cpp_SOURCE_DIR}") + elseif(NOT _jllama_head_count STREQUAL _jllama_tag_number) + message(FATAL_ERROR "llama.cpp pin check: JLLAMA_GIT_TAG is ${JLLAMA_GIT_TAG}, but the fetched commit (${_jllama_fetch_ref}) has commit count ${_jllama_head_count}, i.e. it is b${_jllama_head_count}. With JLLAMA_GIT_COMMIT set, that hash does not resolve the number (run `.github/scripts/llama-next-version.sh --resolve ${JLLAMA_GIT_TAG}`); without it, upstream's tag and its count disagree, which has never happened.") + else() + message(STATUS "llama.cpp pin check: fetched commit count ${_jllama_head_count} = ${JLLAMA_GIT_TAG}") + endif() +else() + message(STATUS "llama.cpp pin check skipped: no git, or ${llama.cpp_SOURCE_DIR} is not a full git checkout") +endif() + # Every ggml source allocating with ggml_aligned_malloc must also release with ggml_aligned_free. # The AMX buffer type did not, which corrupted the Windows heap on AMX-capable CPUs and killed the # process at cleanup; patches/0018 fixes it. Checked here on every platform in every build, @@ -607,15 +663,10 @@ message(STATUS "Backend '${JLLAMA_BACKEND}' - installing files to ${JLLAMA_DIR}" # jllama-build.txt: the build every natives directory came from, read by LlamaLoader before it puts # a GPU module next to a CPU jar's library -- the two must be the same llama.cpp tag (the ggml ABI # between libggml-base and a module is that commit's), and a mismatch is a LlamaException naming -# both, never a silent attempt. The tag is read back from this file's own GIT_TAG line so the -# version bump keeps touching that one line (build-webui and llama-next-version.sh grep it too). -file(READ ${CMAKE_CURRENT_LIST_FILE} _jllama_self) -string(REGEX MATCH "GIT_TAG[ \t]+(b[0-9]+)" _jllama_tag_match "${_jllama_self}") -set(JLLAMA_LLAMA_CPP_TAG "${CMAKE_MATCH_1}") -unset(_jllama_self) -if(NOT JLLAMA_LLAMA_CPP_TAG) - message(FATAL_ERROR "could not read the llama.cpp GIT_TAG (b) from ${CMAKE_CURRENT_LIST_FILE}") -endif() +# both, never a silent attempt. The tag is JLLAMA_GIT_TAG, the pin at the top of the FetchContent +# block -- the b number also when JLLAMA_GIT_COMMIT fetched the commit for it (a module jar and +# a library jar of the same number are the same commit either way, the pin check saw to that). +set(JLLAMA_LLAMA_CPP_TAG "${JLLAMA_GIT_TAG}") if(JLLAMA_MODULE_ONLY) set(_jllama_kind module) else() From a84a9a2754e83331a236314ffc0420279e5abb3e Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 14 Oct 2026 13:39:10 +0000 Subject: [PATCH 3/3] Upgrade llama.cpp from b11617 to b11632 and drop patch 0012 b11632 is master's head of 2026-10-14 (62df1815b), pinned ahead of the tag: JLLAMA_GIT_TAG b11632 with JLLAMA_GIT_COMMIT naming the commit, the first use of the virtual-number pin. Upstream merged the zero-split-sum fallback as ggml-org/llama.cpp#29677 (df18bee11, b11626) on the way; patch 0012 stops applying exactly there -- upstream's line is inside the block it replaced -- so it is dropped, not refreshed, and its runnable guard test_model_split.cpp goes with it: the two functions it linked existed only through the patch, the fixed line sits in a member function with no seam, and the end-to-end route needs two GPU-type devices, which a GPU-less runner cannot provide in one process. The drop note in docs/history/dropped-llama-patches.md records all of that. The other twelve patches apply unchanged at 62df1815b. Besides the fix: cpp-httplib 0.61.0, the RPC transfer logs, --spec-draft-n-ubatch (a new speculative field, not exposed yet), a conv_1d layout fix and a CUDA GDN kernel. Through the production path (FetchContent by commit, the applier, the pin check reporting count 11632 = b11632): ctest 601/601, the model-free native Java tests 27/27 against the linked b11632-62df1815b. Clear JLLAMA_GIT_COMMIT once upstream tags b11632. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_017kVXQfPe1ACLAuk29Y4d2H --- .github/verify-patches-applied.sh | 1 - .github/workflows/publish.yml | 2 +- CLAUDE.md | 20 +- README.md | 2 +- TODO.md | 6 +- docs/history/dropped-llama-patches.md | 34 +++ docs/history/llama-cpp-breaking-changes.md | 2 + llama/CMakeLists.txt | 10 +- ...-split-sum-and-name-the-device-index.patch | 288 ------------------ .../llama/value/LlamaCppVersion.java | 8 +- llama/src/test/cpp/test_model_split.cpp | 126 -------- 11 files changed, 57 insertions(+), 442 deletions(-) delete mode 100644 llama/patches/0012-model-guard-zero-split-sum-and-name-the-device-index.patch delete mode 100644 llama/src/test/cpp/test_model_split.cpp diff --git a/.github/verify-patches-applied.sh b/.github/verify-patches-applied.sh index 418d6dfe..fc66874f 100755 --- a/.github/verify-patches-applied.sh +++ b/.github/verify-patches-applied.sh @@ -17,7 +17,6 @@ # # 0003, 0006, 0007, 0008 -> jllama.cpp / native_server.cpp call the symbols they add, # so dropping one is a compile or link error. -# 0012 -> src/test/cpp/test_model_split.cpp. # 0014 -> src/test/cpp/test_common_log_callback.cpp (link error). # 0001, 0002 -> model-gated Java jobs (Windows argv, LoadProgressCallbackTest). # diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index aa8aa4ca..39bf45b9 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -2337,7 +2337,7 @@ jobs: mvn -q --no-transfer-progress -f llama/pom.xml compile .github/build.sh -DBUILD_TESTING=ON # Every patch has a runnable guard that reds this job if it goes missing (a link error, or - # test_utils.cpp / test_model_split.cpp / test_common_log_callback.cpp). This is the second + # test_utils.cpp / test_common_log_callback.cpp / test_rpc.cpp). This is the second # line: it asserts the applier actually ran and nothing reverted the patched tree, which no # per-patch guard covers directly. Model-free, milliseconds. - name: Verify llama.cpp patches are applied diff --git a/CLAUDE.md b/CLAUDE.md index e16582d0..e46db76b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b11617** +Current llama.cpp pinned version: **b11632** ## Natives jars: library jars and GPU module jars (`.github/natives.csv`) @@ -178,7 +178,7 @@ measurements behind the Windows step are in `docs/handover/local-agent-report-b1 `add_subdirectory(ggml)` when a `ggml` target exists; a module-only build never adds llama.cpp). llama, mtmd and llama-common stay static in `libjllama.so`. **Every ggml default llama.cpp's own CMakeLists changes before its `add_subdirectory(ggml)` is repeated** -- `GGML_LLAMAFILE_DEFAULT` - and `GGML_CUDA_GRAPHS_DEFAULT` at b11617 -- because adding ggml first skips that block; re-check with + and `GGML_CUDA_GRAPHS_DEFAULT` at b11632 -- because adding ggml first skips that block; re-check with `grep -n _DEFAULT CMakeLists.txt` at the new tag on every bump. The second one was missing once: ggml's own default for CUDA graphs is OFF, and a CUDA module built without it lost two thirds of its token generation (265.6 -> 88.8 tg128 on an RTX 3070, prompt processing -20%; Vulkan, which @@ -1107,7 +1107,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network for the asset build; the embed step is plain cmake -P -git clone --depth 1 --branch b11617 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b11632 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build ) mkdir -p webui-generated /tmp/ui-gen cmake -DUI_SOURCE_DIR=/tmp/lc/tools/ui -DUI_BINARY_DIR=/tmp/ui-gen \ @@ -1147,7 +1147,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`JLLAMA_GIT_TAG b11617`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`JLLAMA_GIT_TAG b11632`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1377,15 +1377,14 @@ Current patches: | `0007-server-attach-http-frontend.patch` | **Adds `llama_server_attach(argc, argv, server_context&)`** so the `NativeServer` *attach mode* can serve an **already-loaded `LlamaModel`** over the upstream HTTP frontend — no second model load, no `start_loop()`; the LlamaModel's worker keeps driving the shared `server_context` and the HTTP routes post tasks to its queue (the queue is the synchronization point). Mechanically: (1) extracts the **pure core route table** (`health` … `slots`) out of `llama_server()` into `static void llama_server_register_common_routes(ctx_http, routes)` (shared, so the two entry points cannot drift on the core endpoint set). **Scope note (narrowed at the b10154 bump):** the helper deliberately carries **only** the stable, state-independent route table — **not** the resumable-streaming routes (their handlers differ between router / non-router), the GCP-compat shim, or the experimental **CORS-proxy / MCP-server / built-in-tools** wiring. b10154 (upstream MCP-server support) moved the streaming routes into the middle of that block and coupled tools/CORS to a per-call `server_mcp mcp_mgr` lifecycle, so the earlier contiguous "route-table + CORS-proxy + tools" extraction is no longer possible; `llama_server()` keeps all of that inline, **byte-identical to upstream b10154** (only the route-table block is factored out). (2) adds `llama_server_attach`, which parses only the HTTP-side argv via `common_params_parse`, starts the stream-session GC + `server_http_context`, registers the common route table, the **non-router** resumable-streaming handlers (upstream b10154 paths `/v1/stream` GET/DEL + `/v1/streams/lookup` POST), the GCP-compat shim, and **403 "disabled" stubs for `/cors-proxy` + `/tools`** (attach mode does not wire the experimental CORS-proxy / MCP / built-in-tools host — those belong to a full `llama-server`, not an embedded model), marks ready immediately (model already loaded), and blocks on the HTTP thread until `llama_server_request_shutdown()` — never calling `common_init()`, backend init, `ctx_server.terminate()` or `llama_backend_free()` (the embedding caller owns those). Applies after `0001`+`0006` (same file); closes the "NativeServer — reuse an already-loaded LlamaModel" TODO. Upstream-submittable ("server: let embedding callers attach the HTTP frontend to an existing server_context"). **Refreshed at the b10519 bump:** upstream #26347 dropped the API key from the `/models` + `/v1/models` public-endpoint set and deleted the two trailing `// public endpoint (no API key check)` comments on those route registrations. Those two lines sit inside this patch's route-table removal block, so `git apply` failed ("patch does not apply", `server.cpp:258`) at **every** tag from b10519 on; the fix was to drop the now-wrong comment from all four affected lines (2 on the `-` side, 2 in the extracted helper on the `+` side), keeping the helper byte-identical to the block it replaces. **This is the invariant to re-check on every bump:** the `+` side of `llama_server_register_common_routes()` must stay a verbatim copy of the route table it factors out of `llama_server()`. **Refreshed at the b11104 bump** (upstream #28690, multi-address `--host`): `server_http_context` lost its single `thread` and `listening_address` members in favour of `join()` and a `listening_addresses` vector, one listener thread per bound address. The patch still *applied* cleanly there — only its own `+` lines named the removed members — so the applier could not see it; `llama_server_attach` now logs every address and blocks in `ctx_http.join()`, exactly as upstream's `llama_server()` does. **Refreshed at the b11361 bump** (#29818 adds `POST /v1/systemone` to the route table: one line, added on both sides so the helper stays verbatim) **and at b11401** (#29895 adds a `server_child &` overload of `llama_server()` beside the declarations and a `server_child child;` at the top of the argv entry point, the context of `0007`'s first two hunks; only context moved, the `+`/`-` lines are unchanged). **Changed in 5.2.x, not a refresh: `llama_server_attach(argc, argv, server_context &, server_routes &)` serves the caller's `server_routes`** (`jllama_context::routes`, the one the model load builds) **instead of constructing one on its own frame.** `server_routes`' constructor registers a sleeping-state callback on the model's task queue, `server_queue::on_sleeping_state` only appends, so the frame-local object left a callback into freed memory behind when the attach returned, and the model's next idle sleep (`--sleep-idle-seconds`) ran it -- undefined behaviour, found at the b11361 bump and pinned by `NativeServerAttachSleepIntegrationTest` (model-backed, CI). Consequence: the route handlers read the **model's** params, so the endpoint toggles (`--metrics`, `--props`, `--slots`), `--slot-save-path`, the slot count and the sampling defaults are `ModelParameters` now (`enableMetricsEndpoint()` & co., added for that), and the attach argv configures the HTTP side only. | | `0008-server-models-worker-cmd-override.patch` | **Makes router mode usable in-JVM.** The router (`server-models.cpp`) spawns each model worker by re-executing its own binary (`get_server_exec_path()` = `/proc/self/exe` & friends) — inside a JVM that binary is `java`, not a llama-server, so embedded router workers could never start. The patch adds env `LLAMA_SERVER_WORKER_CMD` (whitespace-split; read in `server_model_meta::update_args`) which replaces only the leading binary-path token of the rendered worker args, letting an embedding host relaunch workers through its own bootstrap — e.g. `java -cp app.jar net.ladenthin.llama.server.NativeServer` (each worker is then a fresh JVM running the classic single-model `NativeServer`). Exposed in Java as `NativeServer.setWorkerCommand(String...)` (JNI `setenv`); exercised by `RouterModeIntegrationTest` (Linux CI). Upstream-submittable (also useful for containerized/wrapped deployments). **Refreshed at the b11429 bump:** #29987 (untagged `4d60b4d08`, the commit before b11429) moved `server_model_meta::update_args` and changed the function after it (`update_caps(const common_params &)`), the hunk's trailing context; the `+` lines are unchanged. Since b11401 (#29895) a worker is a router *child* whose stdout carries the state commands, which is why `LlamaLoader` prints its two diagnostic lines to stderr (a JVM worker's early `System.out` line tripped the router until 2026-10-09, `e9758054`). | | `0006-server-embed-native-server-jni.patch` | **Makes `server.cpp`'s `llama_server` embeddable in the JVM** so the `NativeServer` JNI bridge can run the full upstream HTTP server (WebUI included) inside `libjllama` — see "Two server modes" below. b9870 already exposes `int llama_server(int, char**)` (non-static; no `main` in the file), so the patch only adds embedded-mode support: (1) a `g_llama_server_embedded` flag + `llama_server_set_embedded()` / `llama_server_request_shutdown()` (declared in the committed `src/main/cpp/native_server_bridge.h`); (2) skips installing the process-wide SIGINT/SIGTERM handlers when embedded (they would hijack the JVM's); (3) in embedded mode parses the **forwarded** argv via `common_params_parse` instead of `common_params_parse_main` (whose `GetCommandLineW` recovery would pick up `java.exe`'s command line — the same Windows class of bug `0001` fixes). `llama_server_request_shutdown()` mirrors the SIGTERM path (invokes the installed `shutdown_handler` → `ctx_server.terminate()` unblocks `start_loop()`), giving JNI an out-of-band stop since `ctx_server` is loop-local. **The handler is guarded, and both guards are load-bearing:** upstream's `shutdown_handler` lambdas capture `llama_server()`'s locals (`ctx_http`, `models_routes`, `mcp_mgr`, `ctx_server`) by reference and are never cleared, while `native_server.cpp` signals *repeatedly* until the worker reports it finished (a stop issued before the handler is installed, or before `start_loop()` begins, would otherwise be lost). A request landing after `llama_server()` returned therefore ran the lambda over destroyed objects — a `SIGSEGV` in `server_http_context::stop()` in `RouterModeIntegrationTest.tearDown` (run 36485159455), with router mode's `clean_up()` (unloading workers) making the window wide. The patch (1) takes `g_shutdown_handler_mutex` for every read, write and invocation of the handler (the caller is on another thread) and (2) declares a `llama_server_shutdown_handler_reset` **after** the captured locals, so the handler is cleared before any of them is destroyed; because invocation holds the same lock, that reset waits for an invocation in progress. `0007`'s `llama_server_attach` carries the same guard. Runnable guard: `src/test/cpp/test_native_server_shutdown.cpp`. Applies **after `0001`** (which flips this call site to `common_params_parse_main`), so its context is the post-`0001` tree; regenerate against `0001`+source on a bump. Only touches `tools/server/server.cpp`. | -| `0012-model-guard-zero-split-sum-and-name-the-device-index.patch` | **A GPU that reports zero free memory makes every model load fail with the unactionable `error loading model: vector`.** `llama_model_base::load_tensors` (`src/llama-model.cpp`) weights the per-device layer split by `ggml_backend_dev_memory()`'s `free`, then normalises: `splits[i] /= split_sum`. With a single device reporting `free == 0` that is `0/0` → **NaN** in every split point; NaN compares false against everything, so the `std::upper_bound` below returns the end iterator, `layer_gpu == n_devices()`, and `devices.at(layer_gpu)` throws `std::out_of_range` — whose libc++ `what()` is the bare string `"vector"`, which `llama.cpp`'s `catch (const std::exception &)` prints verbatim. Upstream's `free == 0 && total == 0` host-memory fallback does **not** fire, because `total` is `recommendedMaxWorkingSetSize` and is non-zero. **Reachable since b10618..b10797**: upstream `8c0b9cd04` ("metal : fix memory query under low-memory conditions", [#27701](https://github.com/ggml-org/llama.cpp/pull/27701)) changed `ggml-metal-device.m` to `*free = *total > cur ? *total - cur : 0`; before that clamp an over-committed device (`currentAllocatedSize > recommendedMaxWorkingSetSize`) *underflowed* to a huge `size_t`, which normalised fine, so the same precondition was harmless. That is why the `Java Tests macOS …` jobs went red at the b10792→b10797 step while every Linux/Windows job stayed green — **and why only a GPU build can fail this way at all**: `act_gpu_layers` is `devices.empty() ? 0 : …`, so with no GPU backend `devices` is empty, every layer returns early on `cpu_dev`, and the `.at()` line is unreachable. **Shape:** the two blocks are lifted out of `load_tensors` into free functions declared in `src/llama-model.h`, purely so they can be driven by a test — the failing state needs a real over-committed GPU and cannot be arranged through any public API. `llama_model_splits_normalize()` carries **the fix**: on `split_sum == 0` it `LLAMA_LOG_WARN`s and falls back to an even split (`splits[i] = float(i+1)/splits.size()`), the only neutral choice when no device can be preferred and exactly right for a single device. `llama_model_splits_select_device()` carries **the diagnostic**: it bounds-checks the index and throws a `std::runtime_error` naming the function, the offloaded layer, the device index, the split-point count **and the split points themselves** — with NaN splits that message prints `nan` and names the cause outright, which is precisely what was missing when this had to be diagnosed by reading source. **A second, backend-independent trigger reaches the same line**, found while writing this up and verified against the unfixed library: `--tensor-split` values are parsed with `std::stof` and never range-checked (`common/arg.cpp`), so `-ts 1,-1` cancels out, `split_sum` is 0 again, the split points become `[inf, -nan]`, and every layer maps one past the last device — on CUDA, Vulkan or ROCm just as much as on Metal, with no memory pressure involved. That is what makes this an ordinary upstream defect rather than a Metal edge case, and the warning names both causes rather than only the memory one. Also adds upstream `tests/test-model-split.cpp` (5 cases in upstream's `testing.h` style) + its `llama_build_and_test` registration. Touches `src/llama-model.{cpp,h}`, `tests/test-model-split.cpp` and `tests/CMakeLists.txt` — **none** of which any other patch touches, so it is independent of all of them. **Refreshed at the b11570 bump** (#30299 links each upstream test against only the libraries it uses: `llama_build_and_test` takes `LINK `, and the registration this patch adds reads `llama_build_and_test(test-model-split.cpp LINK llama)` now -- the carried test includes `../src/llama-model.h` and needs `llama` only, like the other internal-function tests in that block; the `src/llama-model.{cpp,h}` and `tests/test-model-split.cpp` hunks are unchanged, only their offsets moved). Upstream-submittable ("model: fall back to an even split when no device reports free memory"); **not yet filed upstream**. **Runnable guard: `src/test/cpp/test_model_split.cpp`** — a FetchContent subproject builds with `LLAMA_BUILD_TESTS=OFF`, so the upstream test above is applied-but-never-compiled here (same as `0001`'s test). That file drives the same two functions from `jllama_test`, which runs on **every** platform in `C++ Tests`, so a bump that drops this patch fails the build at link time everywhere instead of surfacing as one red macOS Java job. **Verification limit — read before assuming this can be dropped:** the *failing path* still cannot be reached without a GPU backend, so the guard pins the arithmetic (what actually broke), not the end-to-end load; the end-to-end proof is the macOS CI job. On a bump, re-check whether upstream added its own `split_sum == 0` guard (grep `split_sum` in `src/llama-model.cpp`) and **drop this patch rather than refreshing it** if they did — the fail-loud applier detects "does not apply", never "upstream already fixed this". **Upstream fixed the defect on 2026-10-14, one tag after this pin:** [ggml-org/llama.cpp#29677](https://github.com/ggml-org/llama.cpp/pull/29677), commit [`df18bee11`](https://github.com/ggml-org/llama.cpp/commit/df18bee11e96557b07aa7b12db0976f3c7ac9165) ("model: split layers evenly when no device reports free memory"), one line in `load_tensors()`: `splits[i] = split_sum != 0.0f ? splits[i] / split_sum : float(i + 1) / n_devices()` -- the same even-split fallback as `llama_model_splits_normalize()` here, without the warning, without the `select_device` diagnostic and **without a test** (the PR touches `src/llama-model.cpp` only). It is **not in b11612** (`10da8c463`, 15 commits before it on master; `src/llama-model.cpp:1648` there is still the bare `splits[i] /= split_sum`, measured), which is why the patch is still carried at this pin and applies unchanged. **At the first bump to a tag that carries `df18bee11` (none up to b11618; b11617 still has the bare division at `src/llama-model.cpp:1648`) the drop-check above fires: drop `0012`, do not refresh it.** What that does to the guard is decided in `TODO.md` ("Drop patch `0012` at the first bump that carries #29677"): `test_model_split.cpp` links the two free functions only this patch adds, so it cannot outlive the patch as it is, and upstream shipped no test to point it at. | | `0014-common-log-callback-sink.patch` | **Gives `common_log` a callback sink: `common_log_set_callback(log, cb, user_data)` (`common/log.{h,cpp}`).** This is what `LlamaModel.setLogger` hooks. Before it, the Java logger was a `llama_log_set()` callback, which has two holes, both found while chasing `slot print_timing` lines interleaving with the Atmosphere agent's streamed answer: **(a)** every model load runs `common_init()`, which re-points `llama_log_set()` at `common_log_default_callback` (`common.cpp:394`), so `setLogger(…)` *before* `new LlamaModel(…)` silently lost the callback; **(b)** the server's own `SRV_*`/`SLT_*` macros are `LOG_INF` and write straight into `common_log`, which `llama_log_set()` never carried, so the per-request `slot …`/`srv …` lines could not be routed to Java at all (the reason `LlamaModelTest#testLogText/JSON` sat `@Disabled` for years). `common_log` upstream offers file, colors, prefix, timestamps, verbosity and JSONL but no hook. The patch adds one: while a callback is set, the worker thread hands every entry to it **instead of** printing to stdout/stderr (a `--log-file` still receives them); the callback gets the bare formatted message (no prefix/timestamp/colors) with the `ggml_log_callback` signature; swapping the callback pauses the worker first, so queued entries reach the *previous* sink (which is what makes `setLogger(format, null)` a synchronous drain). With the sink, `common_init()`'s `llama_log_set()` reset is harmless — it points at the default callback that feeds `common_log`, i.e. exactly the path into the sink — so the ordering problem (a) disappears without any re-install logic, and the `srv`/`slot` lines (b) arrive because they are `common_log` entries. `jllama.cpp`'s `setLogger` therefore sets `common_log_set_callback(common_log_main(), trampoline)` **plus** `llama_log_set(common_log_default_callback)` (so llama/ggml lines feed `common_log` even before the first load). Two consequences to know: the callback runs on `common_log`'s **worker thread**, a plain `std::thread` llama.cpp re-creates on every pause/resume and never attaches to the JVM — the trampoline attaches it **once per worker thread** as a daemon, named `jllama-log-worker`, and a `thread_local` guard detaches it when the thread exits (`get_jni_env_attaching`; `common_log::pause()` joins the worker, a normal exit on which TLS destructors run, and `JNI_OnUnload` pauses it before anything else). Until 5.2.x it attached and detached per line, which created a `java.lang.Thread` object per log line (13 lines of a failed load, 13 `Thread` objects -- `LlamaLoggerTest` pins the single attach now). Each call runs inside a `PushLocalFrame`/`PopLocalFrame` pair, because a thread that stays attached and has no Java frame would otherwise accumulate local references, `setLogger` must call `common_log_set_callback` **outside** `g_log_mutex`, because the pause joins the worker, which needs that mutex to read the callback, and `setLogger` callers are serialized by a **separate** `g_set_logger_mutex`: two unserialized swaps race on the worker's `std::thread` (one joins it while the other assigns a fresh thread over the still-joinable object = `std::terminate`, the whole JVM), which `LlamaLoggerTest#concurrentSetLoggerCallsDoNotRaceOnTheLogWorker` reproduced before the mutex existed. Two caveats the Javadoc carries: a caller must not hold a lock the *previous* callback needs (the drain runs it on the worker while the caller waits), and the verbosity threshold is process-wide and reset by **every** load (`common_params_parse` ends with `common_log_set_verbosity_thold(params.verbosity)`, default 3), so a load without `-lv` puts it back to 3 — a review assumed the opposite, and `LlamaLoggerTest#verbosityThresholdIsProcessWideAndEveryLoadSetsIt` now pins the measured behaviour. And the verbosity threshold applies *before* the sink: at the default (`3`) the Java logger sees errors, warnings and the server's INFO lines, while llama/ggml INFO lines (`common_log_get_verbosity` maps them to TRACE = 4) arrive only from `setLogVerbosity(4)` on — the same filtering the console gets, and a behaviour change for consumers who captured the unfiltered `llama_log_set()` stream before. **Runnable guards:** `src/test/cpp/test_common_log_callback.cpp` (6 tests over a private `common_log_init()` instance: delivery, bare text under prefix+timestamps, clear, swap-drains-to-old-sink, file kept, levels pass through) links the function on every platform, so a bump that drops the patch reds `C++ Tests` at link time; `LlamaLoggerTest` (model-free, needs only `libjllama`: a logger set before a deliberately failing load on a non-GGUF file sees the `srv … loading model` INFO line and llama's ERROR line, in TEXT and JSON) and the re-enabled `LlamaModelTest#testLogText/testLogJSON` plus `#testLoggerSetBeforeLoadSurvivesTheLoad` (vocab-only load) cover the Java side. Upstream-submittable ("common : add a callback sink to common_log for embedding hosts"); **not yet filed upstream**. Touches only `common/log.{h,cpp}`, which no other patch touches. **Refreshed at the b11401 bump** (#29895 adds a `colors` member next to the sink's members and a `get_colors()` next to `set_callback()`; only context moved, the `+`/`-` lines are unchanged, and upstream still has no hook -- it added `common_log_get_colors` only). **Refreshed at the b11612 bump** (`common_log_set_stdo()`, a stdout/stderr on-off switch stored as a `stdo` member right where the sink's members sit, and `print_entry()` now prints to the console only `if (stdo)`): three hunks of `common/log.cpp` lost their context and were regenerated; the sink keeps precedence -- `if (callback) { callback(...) } else if (stdo) { e.print(); }` -- so a sink still replaces the console and `set_stdo(false)` without a sink still silences it. Still no hook upstream (`common/log.h` names only `common_log_default_callback`). **On a bump, check whether upstream added a hook of its own (grep `callback` in `common/log.h`) and, if so, DROP this patch and port `setLogger` to theirs rather than refreshing it.** | | `0015-rpc-embeddable-client-and-stoppable-server.patch` | **Makes ggml-rpc usable inside a JVM** (see "RPC backend" below). Upstream treats every client-side RPC problem as `GGML_ABORT` — a malformed endpoint, a server that is not running, a failed handshake — which in a JVM kills the application over a typo in `--rpc`. **(1)** The *registration* path reports failure instead: `rpc_dispatcher::start()` returns `false`, `try_get_dispatcher()` `nullptr`, `ggml_backend_rpc_get_device_count()` `0`, `ggml_backend_rpc_add_server()` `nullptr`, and `add_rpc_devices()` (`common/arg.cpp`) throws `std::invalid_argument` naming the server where it used to register `nullptr` and silently run without it. Every path *after* registration keeps the original contract through `get_dispatcher()`, which still aborts — a server that vanishes mid-inference stays fatal (TODO). **(2)** `add_server()`'s cache hit is re-checked (a stopped server used to pass registration and abort on the first tensor upload), and `ggml_backend_rpc_get_device_memory()` of a gone server returns 0/0 instead of aborting — RPC devices stay in ggml's process-wide registry forever, and `common_init()` queries the memory of **every** registered device at `-lv 4`, so a stopped server would otherwise take down an unrelated later load. **(3)** The server becomes stoppable: `ggml_backend_rpc_stop_server()` (disconnects the client being served, wakes `accept()` with a loopback connection — the one portable way) and `ggml_backend_rpc_server_listening()`; the loop now reaches its cleanup and frees its backends on every early return, but deliberately **not** `rpc_transport_shutdown()`, whose `WSACleanup()` would invalidate every other RPC socket in the process. **(4)** transport: `MSG_NOSIGNAL`/`SO_NOSIGPIPE`, `socket_t::shutdown()`, and the fds `connect()`/`create_server()`/`accept()` leaked on their error paths are closed. Touches `ggml/include/ggml-rpc.h`, `ggml/src/ggml-rpc/{ggml-rpc.cpp,transport.cpp,transport.h}` and one function in `common/arg.cpp` (`add_rpc_devices`, away from `0001`'s `common_params_parse` hunks). **Runnable guard: `src/test/cpp/test_rpc.cpp`** — links both new functions (a bump that drops the patch fails `C++ Tests` at link time on every platform) and drives registration, stop and the memory query over loopback. Upstream-submittable; **not yet filed upstream**. On a bump, check whether upstream grew its own stop function or non-aborting registration (grep `stop_server` / `GGML_ABORT("Failed to connect` in `ggml-rpc.cpp`). **Refreshed at the b11450 bump** (#26610, `-sm tensor` over RPC, ~800 lines in `ggml-rpc.cpp`, protocol 7 → 8): three hunks lost their context -- new dispatcher members beside `start()`, and new `get_proc_address` names before `return NULL` (the two new names now sit after `ggml_backend_comm_allreduce_tensor`); the `+`/`-` lines are unchanged. The server-to-server comm #26610 adds (a listener on `0.0.0.0`, an `accept()` a stop does not wake) is on file in `TODO.md`. **Refreshed at the b11558 bump** (#29544 turns `GGML_RPC_DEBUG` into a verbosity level 0-3: one `ggml-rpc/log.h` with `LOG_ERROR`/`LOG_WARN`/`LOG_INFO` and `LOG_DBG`/`LOG_DBG2`/`LOG_DBG3`, every `GGML_LOG_*`, `printf` and `fprintf(stderr)` of the backend routed through it, and `transport_name()` where `shutdown()` goes in `transport.h`): 12 of the 24 hunks lost their context and were regenerated against the new text; the changes are the same, and the patch's own five error lines (the HELLO handshake in `negotiate_hello`, four in `rpc_dispatcher::start()`) use `LOG_ERROR` now like the code around them (`LOG_ERROR` is `GGML_LOG_ERROR`). One consequence outside the patch: the server's `Accepted client connection` line is `LOG_INFO` through ggml's log callback now (stderr by default, a `setLogger` sink at verbosity 4), no longer `printf` to stdout; `smoke-rpc.sh` captures both streams into `rpc-server.log`, so its grep is unaffected. | | `0017-ggml-cpu-x86-prefetch-byte-offset-and-msvc-type.patch` | **Makes `GGML_CPU_ALL_VARIANTS=ON` buildable with plain clang on Windows, and fixes a prefetch address that is wrong everywhere.** The four `_mm_prefetch` calls in `ggml/src/ggml-cpu/arch/x86/quants.c` (the SSSE3-without-AVX branch of `ggml_vec_dot_q4_0_q8_0`, and the only `_mm_prefetch` calls in the whole ggml tree) carry two independent defects; both are fixed by routing the address through one new `static inline` helper in the new header `ggml/src/ggml-cpu/arch/x86/prefetch.h`, which is included relatively, so **no CMake change is needed**. **(1) Type -- fatal only on a current clang:** MSVC declares the intrinsic as `void _mm_prefetch(char const *, int)`; clang's own `xmmintrin.h` defines a casting macro but only `#ifndef _MSC_VER`, so on a `*-windows-msvc` target clang leaves the declaration to Microsoft and a typed pointer is `-Wincompatible-pointer-types`. **clang 22 promoted that to an error by default** ([llvm-project #157364](https://github.com/llvm/llvm-project/pull/157364); the [22.1.0 release notes](https://releases.llvm.org/22.1.0/tools/clang/docs/ReleaseNotes.html) add that `-Wno-error=incompatible-pointer-types` downgrades it) -- *not* clang 16, which promoted only `-Wincompatible-FUNCTION-pointer-types`; an earlier version of this row said otherwise. Measured on the real file at `-msse4.2`, unpatched → patched: clang **20.1.8** 4 warnings → 0/0, clang **22.1.3** 4 errors → 0/0, clang **23.1.3** 4 errors → 0/0. Plain clang is exactly what `cmake/x64-windows-llvm.cmake` selects -- and plain clang is the only way to get all **14** CPU variants, because `clang-cl` makes CMake's `MSVC` true and the `if (NOT MSVC)` block in `ggml/src/CMakeLists.txt` then drops `zen4`, `cooperlake`, `sapphirerapids`, `ivybridge` and `piledriver`. **(2) Arithmetic:** `&x[ib] + sizeof(block_q4_0)` adds `sizeof()` *elements*, not bytes -- +324 and +1156 instead of +18 and +34 (the sizes `ggml-common.h` pins by `static_assert`), and +648 instead of +36 for the `2 *` pair. That half is wrong on every platform, including where the call compiles. **Measured:** no throughput difference (see the test-table row), so this is a correctness and portability fix, not a performance one. **Upstream-submittable and meant to be submitted.** **Why upstream's own build is green is measured, not an open question:** their `windows-cpu` release job builds exactly this with plain clang, `GGML_CPU_ALL_VARIANTS=ON` and no suppressing flag (`CMAKE_ARGS` carries only build-target toggles), and ships `ggml-cpu-sse42.dll` -- because the `windows-2025-vs2026` image has **LLVM 20.1.8** on `PATH`, where these are warnings. That is narrower than it looks: the same image also ships **clang 22.1.3** inside Visual Studio 2026 (`VC/Tools/Llvm/x64`), which *fails* on the unpatched file -- upstream reaches the working compiler only because `C:\Program Files/LLVM/bin` comes first on `PATH`. It is **not** a header difference: all three MSVC toolsets on that image (14.29.30133, 14.44.35207, 14.51.36231) declare it identically as `extern void _mm_prefetch(char const*_A, int _Sel);` -- that was the first hypothesis, and a one-off probe workflow (deleted again once it had answered) disproved it by printing the declaration from each toolset on the image. So this patch is **ahead of** upstream rather than diverging: their build breaks the moment the image ships a clang that makes the diagnostic fatal, and until then they compile four warnings and a prefetch address 18x/34x too far. **Runnable guard: `src/test/cpp/test_prefetch.cpp`.** On a bump, run `git grep -n _mm_prefetch ggml/src/ggml-cpu/arch/x86/quants.c` at the new tag: if the call sites already cast **before** the addition, **drop this patch rather than refreshing it**. | | `0016-model-kolibri1.patch` | **Adds Aleph Alpha's Kolibri-1 (architecture `kolibri1`) ahead of upstream** -- a 78B German/English reasoning MoE (3.46B active) that upstream does not support yet ([ggml-org/llama.cpp#29922](https://github.com/ggml-org/llama.cpp/issues/29922), no PR at the time). A **temporary carry, not a fix**: drop it the moment upstream registers the architecture (`git grep -n kolibri src/llama-arch.cpp` at the new tag), and keep `test_kolibri1.cpp` -- its numerical comparisons must then pass against upstream's implementation, while its GGUF-format rows follow whatever format upstream's converter fixes (see the test-table row). **Architecture** (Aleph Alpha's vLLM reference, `aleph_alpha_inference/kolibri1.py` at `049a6a7`): GQA with per-head q/k RMSNorm; `layer_types` interleaves sliding-window layers (NEOX RoPE) with full-attention layers that have **no** positional encoding; sandwich norms around attention and MoE; every layer routed MoE + one ungated shared expert; the router selects top-k on **`logits + expert_bias`** and weights by the **unbiased `sigmoid(logits)`** -- not DeepSeek-V3's sigmoid router (selection on `sigmoid(logits) + bias`), which picks other experts as soon as the bias is non-zero. The graph therefore builds the selection itself and passes it to `build_moe_ffn` (`selected_experts_in`), leaving upstream's shared MoE code untouched. **Derived from two community ports, which are incompatible with each other's GGUFs** -- Eliasfpv28's AFMoE-based one (base b11378, `expert_gating_func` 2, pre-tokenizer `qwen2`) and the Qwen3-MoE-based one behind Hob-forge's GGUFs (base b11381, as carried in [mjwsolo/localcode#101](https://github.com/mjwsolo/localcode/pull/101): gating 5, pre-tokenizer `kolibri1`, rejects gating 2). This patch **loads both dialects**: gating 2, 5 or absent (all mean the one router), `kolibri1` mapped to the Qwen2 pre-tokenizer in `llama-vocab.cpp`, `expert_weights_norm` / `expert_shared_count` / `output.weight` optional. Every source, with license, is listed in the patch header; `REUSE.toml` annotates the file `MIT AND Apache-2.0` (Eliasfpv28's additions are Apache-2.0). Touches `src/llama-arch.{h,cpp}`, `src/llama-model.cpp` (mapping + NEOX rope list), `src/llama-vocab.cpp` (one name), `src/models/models.h` and the new `src/models/kolibri1.cpp`; no converter. **Runnable guard: `src/test/cpp/test_kolibri1.cpp`** (writes tiny random GGUFs in both dialects and compares every logit with an independent double-precision reference). **Verified against the real 78B model on 2026-10-11** (rented 16-vCPU / 123 GB instance, both published dialects fetched on the machine): both load, and a chat turn through the model's own template answers correctly in each -- `content` `"Berlin."` for a German question and `"51"` for 17x3, with the chain of thought in `reasoning_content` and `finish_reason` `stop`. **The two dialects tokenize identically** (`[452 22090 493 1678 2459 5513 46]`, round-tripping to the input sentence), which is the direct evidence that mapping `kolibri1` to the Qwen2 pre-tokenizer is right. Q4_K_M (44 GB) gave 13.4 t/s and Q3_K_S (31 GB) 14.6 t/s of generation on 16 cores. **Two traps for whoever repeats this:** a *raw* completion on this instruct model degenerates into repetition whatever the patch does, so measure through the chat endpoint; and `chatCompleteText` returns only `content`, which is **empty** when a reasoning model spends its token budget inside the thinking block -- use `handleChatCompletions` and a budget in the hundreds. **Still not verified:** GPU backends -- the community ports report real-model checks (top-1 67/67 against a float64 reference for Q8_0, tool calls, German text). | -| `0018-ggml-amx-free-the-aligned-buffer.patch` | **The AMX weight buffer is freed with `free()` but allocated with `_aligned_malloc()` on Windows -- `STATUS_HEAP_CORRUPTION` (0xC0000374) the moment a model is unloaded.** `ggml_backend_amx_buffer_type_alloc_buffer()` (`ggml/src/ggml-cpu/amx/amx.cpp`) allocates with `ggml_aligned_malloc()`, which is `_aligned_malloc()` under `_MSC_VER` (plain clang on a `*-windows-msvc` target defines it too); `ggml_backend_amx_buffer_free_buffer()` called `free()` on it. Fine on Linux (`posix_memalign` + `free`), fatal on Windows, and only there. The patch frees with `ggml_aligned_free(buffer->context, buffer->size)`, exactly what `ggml_backend_cpu_buffer_free_buffer()` does; on Linux that is `free()`, so nothing changes there. **Reached by** the `sapphirerapids` CPU module (the only variant built with `AMX_TILE AMX_INT8`, which `ggml_backend_load_best` picks on Sapphire/Emerald/Granite Rapids; `ggml_amx_init()` is `return true` on `_WIN32`) and a model whose weights have AMX kernels (Q4_0, Q4_1, Q8_0, Q4_K, Q5_K, Q6_K, IQ4_XS -- i.e. every CI model). **Found by the second full run of the modular natives** (38069595101): `Java Tests Windows 2025 x86_64 (clang)` landed on an `Intel(R) Xeon(R) 6973P-C` runner (Granite Rapids), loaded `ggml-cpu-sapphirerapids.dll`, ran `GrammarExample`'s two generations fine and died with exit code -1073740940 at `LlamaModel.close()`; the first run (38065017595) had the same job green on an AMD EPYC 7763 (no AVX-512, `haswell`). **The Windows runners are a CPU lottery**, so which CPU module a Windows job exercises differs per run; before the modular natives the Windows jar was a static Haswell-level build and this code never ran. **Of the six callers of `ggml_aligned_malloc` in the tree this was the only mismatched pair** (`ggml-backend.cpp`, `ggml-cpu.c` twice, `ggml-openvino.cpp`, `ggml-zdnn.cpp` and `ggml.c` all pair correctly), and `ggml-backend.cpp:2464` is the line the fix copies. **Reproduced deliberately rather than waiting for the lottery again**: on a rented Granite Rapids machine, one CPU module staged at a time with the same model and the DLLs CI itself built, `sapphirerapids` crashed **6 of 6** runs -- including at **one** thread, which rules out a data race -- while `cooperlake`, `icelake` and `haswell` were clean 6 of 6, and on Linux with the same CPU all six exit 0 with byte-identical output. The crash lands *after* the inference prints its timings, in cleanup: the signature of a heap corrupted earlier and detected at `free`. With the patch applied the same matrix on the same machine is **clean in all twelve runs**. **The guard is a configure-time check, not a `jllama_test` case**: the AMX code exists only in that one variant of a `GGML_CPU_ALL_VARIANTS` build and executes only on AMX hardware, which the static test build, the local agent's Zen 3 and the cloud sandboxes lack -- so **`llama/cmake/check-aligned-free-pairing.cmake`** fails the configure on **every** platform when a ggml source calls `ggml_aligned_malloc` and never `ggml_aligned_free` (verified red against the unpatched source, naming `amx.cpp`, and green with this patch). A run-time test cannot replace it: without an AMX CPU there is nothing to observe, which is exactly how this shipped. What exercises it: the Windows x86-64 Java test job and the `windows-x86-64` smoke row whenever GitHub hands them an Intel AMX runner; the patch itself is held by `check-patches.py`, applies at b11617, and the patched file compiles with the variant's flags (GCC 13, `-mamx-tile -mamx-int8 -mavx512vnni ...`). Upstream master still carried the `free()` on 2026-10-10; **not yet filed upstream** (nothing before the release). **Drop-check on every bump:** `grep -n "free(buffer->context)" ggml/src/ggml-cpu/amx/amx.cpp` at the new tag -- gone means upstream fixed it, drop the patch. | +| `0018-ggml-amx-free-the-aligned-buffer.patch` | **The AMX weight buffer is freed with `free()` but allocated with `_aligned_malloc()` on Windows -- `STATUS_HEAP_CORRUPTION` (0xC0000374) the moment a model is unloaded.** `ggml_backend_amx_buffer_type_alloc_buffer()` (`ggml/src/ggml-cpu/amx/amx.cpp`) allocates with `ggml_aligned_malloc()`, which is `_aligned_malloc()` under `_MSC_VER` (plain clang on a `*-windows-msvc` target defines it too); `ggml_backend_amx_buffer_free_buffer()` called `free()` on it. Fine on Linux (`posix_memalign` + `free`), fatal on Windows, and only there. The patch frees with `ggml_aligned_free(buffer->context, buffer->size)`, exactly what `ggml_backend_cpu_buffer_free_buffer()` does; on Linux that is `free()`, so nothing changes there. **Reached by** the `sapphirerapids` CPU module (the only variant built with `AMX_TILE AMX_INT8`, which `ggml_backend_load_best` picks on Sapphire/Emerald/Granite Rapids; `ggml_amx_init()` is `return true` on `_WIN32`) and a model whose weights have AMX kernels (Q4_0, Q4_1, Q8_0, Q4_K, Q5_K, Q6_K, IQ4_XS -- i.e. every CI model). **Found by the second full run of the modular natives** (38069595101): `Java Tests Windows 2025 x86_64 (clang)` landed on an `Intel(R) Xeon(R) 6973P-C` runner (Granite Rapids), loaded `ggml-cpu-sapphirerapids.dll`, ran `GrammarExample`'s two generations fine and died with exit code -1073740940 at `LlamaModel.close()`; the first run (38065017595) had the same job green on an AMD EPYC 7763 (no AVX-512, `haswell`). **The Windows runners are a CPU lottery**, so which CPU module a Windows job exercises differs per run; before the modular natives the Windows jar was a static Haswell-level build and this code never ran. **Of the six callers of `ggml_aligned_malloc` in the tree this was the only mismatched pair** (`ggml-backend.cpp`, `ggml-cpu.c` twice, `ggml-openvino.cpp`, `ggml-zdnn.cpp` and `ggml.c` all pair correctly), and `ggml-backend.cpp:2464` is the line the fix copies. **Reproduced deliberately rather than waiting for the lottery again**: on a rented Granite Rapids machine, one CPU module staged at a time with the same model and the DLLs CI itself built, `sapphirerapids` crashed **6 of 6** runs -- including at **one** thread, which rules out a data race -- while `cooperlake`, `icelake` and `haswell` were clean 6 of 6, and on Linux with the same CPU all six exit 0 with byte-identical output. The crash lands *after* the inference prints its timings, in cleanup: the signature of a heap corrupted earlier and detected at `free`. With the patch applied the same matrix on the same machine is **clean in all twelve runs**. **The guard is a configure-time check, not a `jllama_test` case**: the AMX code exists only in that one variant of a `GGML_CPU_ALL_VARIANTS` build and executes only on AMX hardware, which the static test build, the local agent's Zen 3 and the cloud sandboxes lack -- so **`llama/cmake/check-aligned-free-pairing.cmake`** fails the configure on **every** platform when a ggml source calls `ggml_aligned_malloc` and never `ggml_aligned_free` (verified red against the unpatched source, naming `amx.cpp`, and green with this patch). A run-time test cannot replace it: without an AMX CPU there is nothing to observe, which is exactly how this shipped. What exercises it: the Windows x86-64 Java test job and the `windows-x86-64` smoke row whenever GitHub hands them an Intel AMX runner; the patch itself is held by `check-patches.py`, applies at b11632, and the patched file compiles with the variant's flags (GCC 13, `-mamx-tile -mamx-int8 -mavx512vnni ...`). Upstream master still carried the `free()` on 2026-10-10; **not yet filed upstream** (nothing before the release). **Drop-check on every bump:** `grep -n "free(buffer->context)" ggml/src/ggml-cpu/amx/amx.cpp` at the new tag -- gone means upstream fixed it, drop the patch. | | `0019-ggml-cpu-select-the-neon-fp32-to-fp16-conversion.patch` | **Every ARM build converts fp32 to fp16 in software, ours and upstream's.** The `__ARM_NEON` branch of `simd-mappings.h` defines both `GGML_CPU_COMPUTE_*` macros and selects only `FP16_TO_FP32`, so `GGML_CPU_FP32_TO_FP16` falls through to `GGML_COMPUTE_FP32_TO_FP16` -- ggml-base's deliberately portable software bit manipulation -- while `neon_compute_fp32_to_fp16`, one `fcvt`, stays defined and referenced by nothing. The `GGML_CPU_*` pair exists to be the *accelerated* variant; on ARM it was defined to be the unaccelerated one. `__POWER9_VECTOR__` and `__riscv_zfhmin` select both, and the comment on that fallback has always promised NEON does too. **Not a regression** -- absent in all 25 most recent commits touching the file, back to 2025-04-10. **Why only ARM feels it:** `ggml_cpu_fp32_to_fp16`, the `from_float` of `GGML_TYPE_F16`, has vector paths for F16C, RISC-V and s390x (the last from #30157) and **none** for NEON, so there the whole function is the scalar loop through the macro. **In the shipped artifacts:** no `fcvt` of any form in that function in any of this project's 17 ARM libraries (`cpu-linux-aarch64` 8 variants, `cpu-android-aarch64` 7, `cpu-windows-aarch64`, `metal-macos-aarch64`) nor in upstream's own `bin-ubuntu-arm64` / `bin-macos-arm64`; a plain `gcc` with no flags defines `__ARM_NEON` and `__ARM_FP16_FORMAT_IEEE` on aarch64, so nothing can miss it. **Measured 2.45x-13.3x** on five ARM core types, three operating systems and nine compiler builds, with the opposite direction in the same run as a control (within 3.2 % everywhere); **bit-exact over all 2^32 float inputs**, differing only in NaN payload, where x86 F16C gives the same numbers -- so it makes ARM agree with what x86 already ships. **Not an inference speed-up:** `perf` puts 83.9 % of an F16 `pp512` run in tinyBLAS's gemm, which converts in its own microkernel, and 0.03 % here. The patch also moves #30270's two F16C selections from 88 lines below the chain into the `__F16C__` branch (behaviour-preserving, proven at object level) and puts an `#error` pair in their place that refuses a branch defining a COMPUTE macro without selecting it -- its position chosen by a mutation measurement, not by taste (4 of 4 correctness cases caught before the fallback against 3 of 4 directly after the chain). Full record: [`docs/benchmarks/arm-fp16-conversion.md`](docs/benchmarks/arm-fp16-conversion.md). **Submitted upstream as [ggml-org/llama.cpp#30505](https://github.com/ggml-org/llama.cpp/pull/30505)** (open), byte-identical to this diff. **Drop-check on every bump:** `git grep -n "GGML_CPU_FP32_TO_FP16(x) GGML_CPU_COMPUTE_FP32_TO_FP16" ggml/src/ggml-cpu/simd-mappings.h` at the new tag -- if the `__ARM_NEON` branch carries that line, **drop this patch rather than refreshing it**. | -**Dropped patches** (`0004`, `0005`, `0009`, `0010`, `0011`, `0013` -- upstream fixed each defect +**Dropped patches** (`0004`, `0005`, `0009`, `0010`, `0011`, `0012`, `0013` -- upstream fixed each defect itself) are recorded in [`docs/history/dropped-llama-patches.md`](docs/history/dropped-llama-patches.md): why each went, and the runnable guards that stayed behind re-pointed at upstream's contract. Read the matching note before reintroducing a patch for the same defect. **The standing drop-check on every @@ -2409,7 +2408,6 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" | `src/test/cpp/test_jni_helpers.cpp` | 70 | All functions in `jni_helpers.hpp` using a zero-filled `JNINativeInterface_` mock (incl. the `utf8_to_jstring_impl` byte-array string path: emoji byte-preservation, truncated-UTF-8 replace-not-throw). Seven of them pin `jni_guard_impl` — the JNI exception boundary every `Java_*` entry point runs inside — including the `catch (...)` arm that is the only backstop for a non-`std::exception` type, and its two refusals (never `ThrowNew` over a pending Java exception, never with a null class). | | `src/test/cpp/test_tts_wav.cpp` | 2 | The in-memory WAV writer `pcm_to_wav16_bytes` in `tts_wav.hpp` (WAV header/payload + little-endian clamping) — our own code, not upstream. The Qwen3-TTS pipeline it pairs with (`mtmd_helper::gen_audio`) is entirely upstream-owned (no project-side DSP to unit-test here). The load path is additionally covered by `test_tts_params.cpp` (3 tests over `tts_params.hpp`'s `build_tts_params`, plus 2 pinning the upstream `-1` default it depends on), which pins the CPU-thread resolution whose absence used to crash the JVM on every platform — see the `TODO.md` entry for the mechanism. End-to-end coverage is `TtsIntegrationTest`, which is model-gated. | | `src/test/cpp/test_tts_params.cpp` | 13 | The **three** builders every hand-assembled `common_params` goes through: `build_tts_params` (`tts_params.hpp`), `build_train_params` (`train_params.hpp`) and the shared `jllama::resolve_cpu_params` (`cpu_params.hpp`). Each builder is guarded separately on purpose — testing the resolver alone does **not** cover its call sites, because `train_engine.cpp` is compiled into `jllama` only, never into `jllama_test`, and `LlamaTrainerIntegrationTest` is gated on `net.ladenthin.llama.train.model`, which no CI job sets. Without these the JVM-abort bug could regress in the trainer on every platform, unseen. | -| `src/test/cpp/test_model_split.cpp` | 7 | The two `load_tensors()` split helpers that `patches/0012` extracts out of llama.cpp's `src/llama-model.cpp` — `llama_model_splits_normalize` (proportional split, single device, and the zero-sum case that used to produce NaN, **and the cancelling `--tensor-split` case** — `-ts 1,-1` reaches the identical line on any backend with no GPU memory pressure at all) and `llama_model_splits_select_device` (every layer maps to a real device index; malformed split points throw a message that names the function, the layer, the index and the split values instead of libc++'s bare `"vector"`). **This is the runnable guard for `0012`**: the patch also ships an upstream `tests/test-model-split.cpp`, but a FetchContent subproject builds with `LLAMA_BUILD_TESTS=OFF`, so that one is applied-but-never-compiled here. This file is the only place the two functions are linked in CI, on every platform — so a bump that drops the patch fails the `C++ Tests` build outright rather than resurfacing as one red macOS Java job. It is the one test file that includes an **internal** upstream header (`llama-model.h`, via the `${llama.cpp_SOURCE_DIR}/src` include dir added for it), which is deliberate: a signature drift should fail loudly at compile time. | | `src/test/cpp/test_prefetch.cpp` | 5 | **The runnable guard for `patches/0017`** (the x86 prefetch helper). The patch fixes two defects in the four `_mm_prefetch` calls of `ggml_vec_dot_q4_0_q8_0`'s SSSE3-without-AVX branch, and only one of them is a build error. The **type** half is caught by the compiler, but only by a recent one: plain clang on a `*-windows-msvc` target gets MSVC's `char const *` declaration (its own casting macro sits behind `#ifndef _MSC_VER`), and the resulting `-Wincompatible-pointer-types` is an **error by default from clang 22 on** -- measured on the real file at `-msse4.2`: 4 warnings on 20.1.8, 4 errors on 22.1.3 and 23.1.3, and 0/0 on all three once patched (`-msse4.2 -mavx` compiles either way, taking the AVX branch). So `GGML_CPU_ALL_VARIANTS=ON` stops building with plain clang on Windows once the compiler is new enough -- via the **`sse42`** variant, the only one that reaches `#elif defined(__SSSE3__)` (the `x64` baseline has no SSSE3 and takes the scalar branch; everything richer has AVX). Nothing in a test can cover that half. The **arithmetic** half is what this file guards, and it is silent: `&x[ib] + sizeof(block_q4_0)` adds `sizeof()` *elements*, so with the sizes `ggml-common.h` pins by `static_assert` (18 and 34) the prefetch targets +324 and +1156 bytes instead of +18 and +34, and the `+ 2 *` pair +648 instead of +36 -- wrong on **every** platform, including the ones where the call compiles. `ggml_prefetch_at()` casts to `const char *` first, which is what makes the offset byte-wise. **The case that makes the file worth having:** writing the cast around the whole expression, `(const char *)(&x[ib] + sizeof(block_q4_0))`, silences the compiler and keeps the wrong address -- that form builds and fails here. Verified both ways: 5/5 green against the patched header, 5/5 red against a deliberately element-wise one. Includes two internal headers (`ggml-common.h` for the block types, `prefetch.h` for the helper) via the `${llama.cpp_SOURCE_DIR}/ggml/src` and `.../ggml/src/ggml-cpu/arch/x86` include dirs added for it. **Measured, so the upstream report does not overclaim:** the wrong prefetch distance costs no measurable throughput (Ryzen 7 5800H forced onto the `sse42` kernel, 0.6B Q4_0: pp512 101.3 vs 100.9, tg128 37.3 vs 37.0, inside each other's error bars) -- it is a correctness and portability fix, not a performance one. | | `src/test/cpp/test_kolibri1.cpp` | 5 | **The runnable guard for `patches/0016`** (Kolibri-1). Writes tiny random `kolibri1` GGUFs with the public `gguf` API, loads them through the real library on the CPU and compares every logit -- one batch, and token by token through the iSWA KV cache -- with an independent double-precision reference written from Aleph Alpha's vLLM implementation, not from the patch. The layer pattern puts a full-attention (NoPE) layer between sliding ones, the sequence is longer than the window, and the expert biases are large enough that DeepSeek-V3's router would pick other experts (one test asserts that, so the comparison cannot pass vacuously). Covers both published GGUF dialects (gating 2 + `qwen2` + `output.weight`; gating 5 + `kolibri1` + tied output), renormalized routing, and the rejection of another gating function. **What happens when upstream supports Kolibri-1 and `0016` is dropped** -- the file compiles unchanged (public `llama.h`/`gguf.h`/`ggml.h` only, nothing from the patch), but its tests split in two. The **numerical comparison** against the reference must stay green, and `TheDeepSeekRouterWouldGiveDifferentLogits` never touches the library at all; a red comparison means upstream computes something else than Aleph Alpha's reference (router, window boundary, positional encoding) and is a finding, not a test to adjust. The **GGUF format** each test writes -- architecture name, key and tensor names, `expert_gating_func`, pre-tokenizer, optional `output.weight` -- is upstream's converter's decision, not ours: `AfmoeDialectMatchesTheReference` (gating 2) is the likeliest to go red if upstream follows the gating-5 port, `RenormalizedRoutingMatchesTheReference` writes no gating key at all, and `AnotherGatingFunctionIsRejected` assumes a rejection upstream need not make. A red format row is a real signal as well -- the published GGUFs of that dialect stop loading without the patch -- so decide it deliberately (a small compatibility patch, or document that those files must be reconverted) and only then move the row to upstream's format: one `{gating, pre, ...}` entry per test, with the reference untouched. | | `src/test/cpp/test_model_flags.cpp` | 4 | **The contract between the Java CLI-flag registries and llama.cpp's server argument parser.** CMake reads `ModelFlag.java` + `ModelOption.java` (`cmake/extract-java-wire-names.cmake` → a generated header of `{name, contract}` pairs), and this file asserts every `SERVER_PARSER` name is in `common_params_parser_init(params, LLAMA_EXAMPLE_SERVER).options`. It exists because **no Java test can catch this class**: `ModelFlagTest`/`ModelParametersExtendedTest` pin the *string mapping* (`hasKey("--mlock")`), never that llama.cpp still accepts the string, so they stay green forever while the flag is dead — and `common_params_parse` treats an unregistered option as a hard error, so the affected builder method makes the model **unloadable**, not merely ineffective. **A grep over `arg.cpp` is not a substitute**: `--grp-attn-n`/`-w` are present there at every pinned tag but `set_examples()`-scoped to `LLAMA_EXAMPLE_COMPLETION`/`PASSKEY`, so the server parser rejects them exactly like a deleted flag — only the real option table sees that. `--vocab-only` is the one exemption, and it declares itself `CliContract.PROJECT_PSEUDO` on its own constant rather than appearing in a list inside this file; the test asserts such a name is **still unknown** to the parser (an exemption upstream later registers would be hiding a real check) and that the exempt set is non-empty. | @@ -2418,11 +2416,11 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" | `src/test/cpp/test_native_server_shutdown.cpp` | 3 | **The runnable guard for the shutdown-handler guard in `patches/0006`/`0007`.** Runs the real `llama_server()` in router mode over an empty `--models-dir` on an ephemeral loopback port (no model, no worker) and stops it the way `native_server.cpp` does. Pins a clean stop (exit code 0), that `llama_server_request_shutdown()` **after** the server returned is a no-op (the deterministic form of the CI `SIGSEGV` — it crashed every run before the fix), and that requests hammered from another thread while the server tears down are safe (5 rounds). Compiled on non-Android only, like `server.cpp` itself. | -**Current total: 608 tests (all passing).** +**Current total: 601 tests (all passing).** #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `JLLAMA_GIT_TAG b11617`. +llama.cpp is fetched via CMake FetchContent, pinned to `JLLAMA_GIT_TAG b11632`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.18.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the @@ -2464,7 +2462,7 @@ so missing stubs are caught immediately rather than silently. - Upstream result type `to_json()` → `test_server.cpp` - `utils.hpp` function or upstream utility → `test_utils.cpp` - A function one of the local `patches/` adds to llama.cpp → its own file, e.g. - `test_model_split.cpp` for `0012`. Give every patch that introduces a callable a + `test_prefetch.cpp` for `0017`. Give every patch that introduces a callable a guard here: upstream tests carried by a patch are **not** compiled (a FetchContent subproject sets `LLAMA_BUILD_TESTS=OFF`), so this is the only place a dropped patch reds CI on every platform instead of on whichever job happens to hit it. diff --git a/README.md b/README.md index c854cab0..b9a912d5 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b11617](https://img.shields.io/badge/llama.cpp-%23b11617-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b11617) +[![llama.cpp b11632](https://img.shields.io/badge/llama.cpp-%23b11632-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b11632) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/TODO.md b/TODO.md index 5f421c27..e4199a6a 100644 --- a/TODO.md +++ b/TODO.md @@ -842,7 +842,9 @@ whose "17 issues found" status once posted on PRs #248/#298; it posts nothing on never set up deliberately. Whether the project wants one, with which policy, is the owner's call; nothing here blocks on it. -### Drop patch `0012` at the first bump that carries #29677 (upstream merged the fix on 2026-10-14) +### ~~Drop patch `0012` at the first bump that carries #29677~~ -- done at the b11632 bump + +**Done at b11632** (the first number carrying the fix is b11626, `df18bee11`; the bump went to master's head of the day, b11632, pinned through `JLLAMA_GIT_COMMIT` ahead of the tag). The applier fired as predicted in reverse: `0012` does **not** apply at `df18bee11` (upstream's line sits inside the block the patch replaces), so the drop was forced, not silent. The guard went with it -- see the `0012` note in `docs/history/dropped-llama-patches.md` for why no replacement exists. The text below is the plan as it stood, kept for the reasoning. **The drop-check of `0012` fires at the first bump to a tag that contains `df18bee11` -- not at b11612, and not at b11617 either** (checked: no tag up to b11618 carries it; b11617 still divides by `split_sum` unguarded at `src/llama-model.cpp:1648`). On 2026-10-14 upstream merged [ggml-org/llama.cpp#29677](https://github.com/ggml-org/llama.cpp/pull/29677), commit @@ -936,7 +938,7 @@ treat an unknown flag as pass-through). So the measurement stands as recorded an to add this case to one of those issues, or wait. Re-check on the next sccache release before spending a CI run on it. -There are **thirteen** patches today (`0001`-`0003`, `0006`-`0008`, `0012`, `0014`-`0019`). Each +There are **twelve** patches today (`0001`-`0003`, `0006`-`0008`, `0014`-`0019`). Each accepted upstream change deletes a patch from the bump checklist once the pin is bumped past it. - **Submittable as they are** (self-contained, no jllama specifics): `0001` (Windows arg-parse embed diff --git a/docs/history/dropped-llama-patches.md b/docs/history/dropped-llama-patches.md index 9dcb7904..1107416a 100644 --- a/docs/history/dropped-llama-patches.md +++ b/docs/history/dropped-llama-patches.md @@ -11,6 +11,40 @@ themselves. Moved here from `CLAUDE.md`, which lists only the patches still carr records why the patch went and what guard, if any, stays behind -- read the matching note before reintroducing a patch for the same defect. +**`0012` was dropped at the b11632 bump** (the first number that carries the fix is b11626). Upstream merged +[ggml-org/llama.cpp#29677](https://github.com/ggml-org/llama.cpp/pull/29677) (commit +[`df18bee11`](https://github.com/ggml-org/llama.cpp/commit/df18bee11e96557b07aa7b12db0976f3c7ac9165), +"model: split layers evenly when no device reports free memory", 2026-10-14, first tagged at +**b11626**): one line in `llama_model_base::load_tensors()`, +`splits[i] = split_sum != 0.0f ? splits[i] / split_sum : float(i + 1) / n_devices();` -- the same +even-split fallback `0012` carried in `llama_model_splits_normalize()`, for the same two triggers (a +device reporting zero free memory, the over-committed Metal case of b10797, and a `--tensor-split` +that cancels out, `-ts 1,-1`, on any backend). **Here the applier did fire**, unlike for `0010`: the +fixed line sits inside the block `0012` lifted out of `load_tensors()`, so the patch stopped applying +at exactly the commit that made it redundant. Dropped, not refreshed. + +**What went with it, deliberately:** the `LLAMA_LOG_WARN` naming the cause (upstream splits evenly +without a word), the bounds check in `llama_model_splits_select_device()` whose message named the +layer, the device index and the split points (upstream keeps `devices.at()`, so a malformed split +still reads as libc++'s bare `"vector"`), and the never-compiled upstream `tests/test-model-split.cpp`. +None of that is worth a patch on its own. One input upstream's line does **not** cover, for the +record: a NaN weight (`-ts nan,1` -- `std::stof` accepts it, and `NaN != 0.0f` is true) still produces +NaN split points and the bare `"vector"`. If that ever matters it is a separate upstream issue. + +**The runnable guard went with the patch, and no guard replaced it -- the one drop note in this file +that says so.** `src/test/cpp/test_model_split.cpp` drove `llama_model_splits_normalize` and +`llama_model_splits_select_device`, two free functions that existed only through the patch, so it +could not link without it; and the fixed line is inside a member function with no seam, so there is +nothing upstream to point a unit test at. The end-to-end route (`llama_model_load` with two GPU-type +devices and `-ts 1,-1`) was examined and is not reachable from `jllama_test` on a GPU-less runner: it +needs **two** devices of type GPU, and the only in-process source of one is the RPC backend, whose +server state is global (`0015`'s `g_rpc_server_*`), so a second loopback server cannot coexist with +the first. What remains possible is a CI smoke on the `linux-x86-64` row -- two `RpcServer` JVMs and +a third load with `--tensor-split 1,-1` -- one platform, one row, and not taken: the fix is one line +of upstream's that every GPU build now exercises on every load, and the macOS Java test jobs, where +the defect was found, run it on Metal every time. Should the line regress, that is where it shows, +as it did at b10797. + **`0010` was dropped at the b11080 bump.** Upstream merged [ggml-org/llama.cpp#28518](https://github.com/ggml-org/llama.cpp/pull/28518) ("json: Fixed json enum handling", first tagged at **b11080**), which fixes the defect at its diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index ab01cfb9..b924be4b 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -890,3 +890,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b11574–b11612 | patches + upstream verification | **Three of the thirteen patches needed a refresh, none was dropped.** `0001`: `tests/test-state-restore-fragmented.cpp` is gone (#30023 consolidated the context-state tests into `tests/test-llama-context.cpp`), so its hunk was dropped (36 → 35 `diff --git` headers); the consolidated test has the `--models` / `filtered_argv` `main()` and by the patch's own rule keeps `common_params_parse`. `0014`: #30023 also added `common_log_set_stdo()` and a `stdo` member exactly where the sink's members and `print_entry()`'s console call sit -- three hunks regenerated, the sink keeps precedence (`if (callback) ... else if (stdo) e.print()`), pinned by a seventh case in `test_common_log_callback.cpp`. `0006` failed only in the cascade behind `0001` (its context is the post-`0001` tree) and applies unchanged once `0001` does. The other ten apply unchanged. `check-patches.py`: 13 patches, 101 hunks, 0 problems; the real applier on a fresh `FetchContent` checkout of b11612 (stamp head `10da8c463`, 57 modified files plus the stamp), then the static test build, `ctest` **608/608**, and the model-free native Java tests over the library (`NativeLibraryLoadSmokeTest` with the `LlamaCppVersion` pin against the linked `b11612-`, `LlamaLoggerTest`, `RpcServerTest`, `BackendLoadTest`: 27/27). **Drop-checks against pristine b11612, and the one that will fire next:** `src/llama-model.cpp:1648` is still the bare `splits[i] /= split_sum` (`0012`) -- **but upstream merged the fix the same day, 15 commits past b11612**: [#29677](https://github.com/ggml-org/llama.cpp/pull/29677) / `df18bee11` ("model: split layers evenly when no device reports free memory"), the identical even-split fallback and no test. The first bump past it drops `0012`; the plan for its guard `test_model_split.cpp`, which links the two functions only the patch provides, is in `TODO.md` ("Drop patch `0012` at the first bump that carries #29677"). The rest unchanged: no `kolibri` under `src/` (`0016`); `common/log.h` has a `set_stdo` but no hook, only `common_log_default_callback` (`0014`); no `stop_server`, the connect failure still aborts at `ggml-rpc.cpp:645` (`0015`); the four `_mm_prefetch` calls at `quants.c:846-865` still uncast (`0017`); `amx.cpp:46` still `free(buffer->context)` (`0018`); the `__ARM_NEON` branch of `simd-mappings.h` still selects the software conversion (`0019`); `arg.cpp:1288` still carries the `GetCommandLineW` override and no `common_params_parse_main` (`0001`); `server-context.cpp:1212` still assigns the load-progress callback unconditionally (`0002`); no `LLAMA_SERVER_WORKER_CMD` (`0008`), no `get_slot_prompt_similarity` (`0003`). `check-natives.py`: 23 natives jars, 0 disagreements; spotless clean. Not run locally: the model-backed Java suite (no GGUF in the sandbox) and the modular build -- CI's test and smoke jobs cover both. | | b11612–b11617 | 5 commits, 5 files, ~99/85 lines (9 KiB; no `tools/ui`, no `.github`, no `vendor`). Four are Vulkan (#29069 bf16 activations for mulmat, #30431 the shader count for Intel Arc Pro B65/B70, #29835 a workaround for a device reporting free memory above total on Intel Windows, #30495 a compile fix) -- the module jar `vulkan-*` rebuilds them, nothing here reads them. One is `src/models/` (#30480 "llama : build the k-pool selection in the causal mask layout"): `glm5-next.cpp` and `qwen4exp.cpp` hand the DSA/QSA selection to `build_attn_mha` in the mask's own 4-D layout instead of reshaping it, with `ggml_are_same_shape` asserts -- graph-internal to two architectures, outside every patch-target file. Nothing on the priority list changed; `src/llama-model.cpp` is untouched (`split_sum` still unguarded at `:1648`, `0012` still carried -- `df18bee11` / #29677 is in no tag up to b11618). Toolchain pins unchanged. **No project source change.** | | b11612–b11617 | patches + upstream verification | **All thirteen patches apply unchanged at b11617** (`git apply` in order on a pristine b11617 worktree, 57 modified files; `check-patches.py`: 13 patches, 101 hunks, 0 problems; the real applier on a fresh `FetchContent` checkout -- `rm -rf build` -- with stamp head `14310190c`), then the static test build, `ctest` **608/608**, and the model-free native Java tests over the library (`NativeLibraryLoadSmokeTest` with the `LlamaCppVersion` pin against the linked `b11617-`, `LlamaLoggerTest`, `RpcServerTest`, `BackendLoadTest`: 27/27). Drop-checks against pristine b11617 are those of the b11612 row, the range touching no patch-target file: `split_sum` unguarded at `src/llama-model.cpp:1648` (`0012`; the upstream fix `df18bee11` / #29677 is still in no tag -- newest b11618 -- so the drop rule in `TODO.md` keys on the fixing commit now, not on "past b11612"), no `kolibri` (`0016`), no `common_log` hook (`0014`), no `stop_server` (`0015`), uncast `_mm_prefetch` (`0017`), `free(buffer->context)` (`0018`), the software fp16 conversion on `__ARM_NEON` (`0019`), the `GetCommandLineW` override and no `common_params_parse_main` (`0001`), the unconditional load-progress assignment (`0002`), no `LLAMA_SERVER_WORKER_CMD` (`0008`), no `get_slot_prompt_similarity` (`0003`). `check-natives.py`: 23 natives jars, 0 disagreements. Not run locally: the model-backed Java suite and the modular build -- CI covers both. | +| b11617–b11632 | 15 commits (9 to `df18bee11` = b11626, 6 more to master's head of 2026-10-14, `62df1815b` = b11632 -- pinned through `JLLAMA_GIT_COMMIT` ahead of the tag), 43 files, ~1,250/180 lines (no `tools/ui`, no `.github/workflows/release.yml`, root `CMakeLists.txt` unchanged). **`src/llama-model.cpp` (#29677, `df18bee11`): the one-line even-split fallback in `load_tensors()` -- the defect `patches/0012` carried, fixed upstream; see the patches row.** b11626–b11632 on top: `common/common.h` + `common/arg.cpp` (#30459 `--spec-draft-n-ubatch`, a new `speculative.draft.n_ubatch` field and server option -- additive, not exposed in `ModelOption` yet, and away from `0001`'s hunks, which apply unchanged), `ggml/include/ggml.h` (#30501, a comment on `mul_mat_id` duplicate ids), `ggml/src/ggml.c` + `tests/test-conv-1d.cpp` (#28738 conv_1d result layout for batched input -- a graph-level fix in the CPU path, no API change), `ggml-cuda` (#29353 GDN chunked kernel, `gated-delta-net-mma.cu` new: the CUDA module jar), a code-review skill and docs. `vendor/cpp-httplib/` (#30494, cpp-httplib 0.61.0: `httplib.{h,cpp}`, +111/-34; `BORINGSSL_VERSION` still `0.20260929.0`) -- `jllama` links the `cpp-httplib` target, the server and the download code recompile against it, nothing here names its internals. `ggml/src/ggml-rpc/` (#30481 "improve RPC tensor transfer logs": a `rpc_log` helper with its own `log.cpp`, the transfer sizes at `LOG_DBG2`, two `LOG_WARN`) -- `0015` applies unchanged, its hunks sit outside the logging lines. `common/jinja/value.cpp` (#30362, floats in `tojson` written as Python does), `ggml-cuda` (#30325 SIMT conv2d on AMD, #28195 RDNA3.5 MMQ tiles), `ggml-opencl` (#30365, #30418: Adreno MoE), six `src/models/*` embedding graphs (#30461 drops the `t_embd` expand). Nothing on the priority list changed (`common/common.h`, `chat.h`, `arg.h`, `llama.h`, `log.h`, `tools/server/*.h`, `mtmd*.h`, `ggml/include` all untouched); toolchain pins unchanged. **No project source change** beyond the guard that leaves with `0012` (`test_model_split.cpp`, its CMake line and the `${llama.cpp_SOURCE_DIR}/src` include dir only it needed). | +| b11617–b11632 | patches + upstream verification | **`0012` dropped, twelve patches apply unchanged** (`git apply` in order on pristine `df18bee11` and `62df1815b` worktrees, 54 modified files; `check-patches.py`: 12 patches, 95 hunks, 0 problems). **The drop-check fired both ways:** `src/llama-model.cpp:1650` is the guarded `splits[i] = split_sum != 0.0f ? ... : float(i + 1) / n_devices()` now, and `git apply --check` of `0012` on the `df18bee11` tree fails at `src/llama-model.cpp:1638` -- upstream's line is inside the block the patch replaced, so unlike `0010` this one could not have been carried on by mistake. **The first bump pinned ahead of the tag**: no tag past b11618 existed on 2026-10-14, so `JLLAMA_GIT_TAG b11632` is paired with `JLLAMA_GIT_COMMIT 62df1815b…` and the production path ran exactly as CI will -- `FetchContent` by commit, the applier (stamp head `62df1815b`), and the pin check reporting `fetched commit count 11632 = b11632` (`git rev-list --count`, the number upstream's `release.yml` stamps the tag with and the one llama.cpp's own `build-info` reports). Then the static test build, `ctest` **601/601** (608 minus the seven of `test_model_split.cpp`), and the model-free native Java tests over the library (`NativeLibraryLoadSmokeTest` with the `LlamaCppVersion` pin `b11632` against the linked `b11632-62df1815b`, `LlamaLoggerTest`, `RpcServerTest`, `BackendLoadTest`: 27/27). The same chain had run at `df18bee11` / b11626 first (601/601, 27/27, `b11626-df18bee11`). The other drop-checks against pristine `62df1815b`: no `kolibri` (`0016`), no `common_log` hook (`0014`), no `stop_server` (`0015`), uncast `_mm_prefetch` (`0017`), `free(buffer->context)` (`0018`), the software fp16 conversion on `__ARM_NEON` (`0019`), the `GetCommandLineW` override and no `common_params_parse_main` (`0001`), the unconditional load-progress assignment (`0002`), no `LLAMA_SERVER_WORKER_CMD` (`0008`), no `get_slot_prompt_similarity` (`0003`). `check-natives.py`: 23 natives jars, 0 disagreements; spotless clean. Not run locally: the model-backed Java suite and the modular build -- CI covers both. When upstream publishes `b11632`, clear `JLLAMA_GIT_COMMIT` (`llama-next-version.sh` prints the NOTE); nothing else moves. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index f8a900cf..a7ee88b0 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -344,8 +344,8 @@ endif() # hash, every other consumer keeps the number, and the pin check after FetchContent_MakeAvailable # asserts that the fetched commit's count IS the number. Clear it once the tag exists; # `llama-next-version.sh` says when, and `llama-next-version.sh --resolve b` prints the hash. -set(JLLAMA_GIT_TAG b11617) -set(JLLAMA_GIT_COMMIT "") +set(JLLAMA_GIT_TAG b11632) +set(JLLAMA_GIT_COMMIT "62df1815bc053066761f4eac58e288f23778e164") # until upstream tags b11632 if(NOT JLLAMA_GIT_TAG MATCHES "^b[0-9]+$") message(FATAL_ERROR "JLLAMA_GIT_TAG must be a llama.cpp release number b, got '${JLLAMA_GIT_TAG}'") endif() @@ -1099,7 +1099,6 @@ if(BUILD_TESTING) src/test/cpp/test_common_log_callback.cpp src/test/cpp/test_tts_wav.cpp src/test/cpp/test_tts_params.cpp - src/test/cpp/test_model_split.cpp src/test/cpp/test_prefetch.cpp src/test/cpp/test_kolibri1.cpp src/test/cpp/test_model_flags.cpp @@ -1123,11 +1122,6 @@ if(BUILD_TESTING) # jni.h / jni_md.h needed by jni_helpers.hpp (mock JNI tests, no JVM required) ${JNI_INCLUDE_DIRS} ${llama.cpp_SOURCE_DIR}/tools/server - # llama.cpp's internal src/ -- test_model_split.cpp includes llama-model.h to drive the - # two load_tensors() split helpers that patches/0012 extracts. Upstream's own tests do the - # same (tests/test-batch-alloc.cpp, tests/test-quantize-stats.cpp include "../src/..."). - # No header name in that directory collides with tools/server, tools/mtmd or src/main/cpp. - ${llama.cpp_SOURCE_DIR}/src # ggml's internal src/ and the x86 CPU-backend directory -- test_prefetch.cpp includes # ggml-common.h (for block_q4_0 / block_q8_0, whose sizes the prefetch offsets depend on) # and prefetch.h, the helper patches/0017 adds. Same reasoning as the line above: the diff --git a/llama/patches/0012-model-guard-zero-split-sum-and-name-the-device-index.patch b/llama/patches/0012-model-guard-zero-split-sum-and-name-the-device-index.patch deleted file mode 100644 index df9ff08f..00000000 --- a/llama/patches/0012-model-guard-zero-split-sum-and-name-the-device-index.patch +++ /dev/null @@ -1,288 +0,0 @@ -model: fall back to an even split when no device reports free memory, and name the device index -on a bad lookup (the two load_tensors() blocks lifted into free functions so they can be tested). - -STATUS 2026-10-14: FIXED UPSTREAM -- DROP THIS PATCH AT THE FIRST llama.cpp BUMP TO A TAG THAT CARRIES df18bee11. -ggml-org/llama.cpp#29677, commit df18bee11 ("model: split layers evenly when no device reports -free memory"), https://github.com/ggml-org/llama.cpp/commit/df18bee11e96557b07aa7b12db0976f3c7ac9165, -puts the same even-split fallback into load_tensors(): - - splits[i] = split_sum != 0.0f ? splits[i] / split_sum : float(i + 1) / n_devices(); - -It is NOT in b11612 (10da8c463, 15 commits earlier on master) nor in b11617, so the patch is still -needed at those pins and applies unchanged. Drop-check: `git grep -n split_sum src/llama-model.cpp` at the new tag -- -the guarded line means DROP, not refresh (the patch keeps applying next to it, so the applier will -not notice). Upstream ships no test. The runnable guard src/test/cpp/test_model_split.cpp links the -two functions this patch adds and cannot outlive it as it is; what replaces it is decided in -TODO.md, "Drop patch 0012 at the first bump that carries #29677". - -diff --git a/src/llama-model.cpp b/src/llama-model.cpp -index 6872589fa..a56e4bb04 100644 ---- a/src/llama-model.cpp -+++ b/src/llama-model.cpp -@@ -1559,6 +1559,75 @@ void llama_model_base::load_vocab(llama_model_loader & ml) { - vocab.load(ml, kv); - } - -+// Turn the per-device weights collected in `splits` (each device's free memory, or the values -+// the user passed to --tensor-split) into cumulative, normalized split points in (0, 1], so that -+// splits[i] is the fraction of the offloaded layers that belongs to devices [0, i]. -+// -+// Split out of load_tensors() so it can be unit-tested: the degenerate all-zero case below is -+// only reachable with a real over-committed GPU, which a test cannot arrange through the API. -+void llama_model_splits_normalize(std::vector & splits) { -+ // sum and normalize the splits to get the split points -+ float split_sum = 0.0f; -+ for (size_t i = 0; i < splits.size(); ++i) { -+ split_sum += splits[i]; -+ splits[i] = split_sum; -+ } -+ -+ if (split_sum == 0.0f) { -+ // The weights sum to zero, so there is nothing to weight the layers by. Dividing here would -+ // put a NaN (or an inf) in the split points, and NaN compares false against everything, so -+ // the std::upper_bound in llama_model_splits_select_device() below would return the end -+ // iterator and the lookup would run off the end of the device list. -+ // -+ // Two different inputs reach this, and both are reachable in practice: -+ // -+ // * a device that reports no free memory -- e.g. a Metal device whose currentAllocatedSize -+ // has grown past its recommendedMaxWorkingSetSize reports free == 0 with a non-zero -+ // total, so the `free == 0 && total == 0` host-memory fallback in the caller does not -+ // fire and every weight is 0; -+ // * an explicit --tensor-split whose values cancel out, e.g. `-ts 1,-1`. The values are -+ // parsed with std::stof and never range-checked, so this reaches us on any backend. -+ // -+ // Split evenly instead -- the only neutral choice when no device can be preferred over -+ // another, and exactly right for a single device. -+ LLAMA_LOG_WARN("%s: layer split weights sum to zero " -+ "(no device reported free memory, or --tensor-split cancels out), " -+ "splitting layers evenly\n", __func__); -+ for (size_t i = 0; i < splits.size(); ++i) { -+ splits[i] = float(i + 1)/splits.size(); -+ } -+ return; -+ } -+ -+ for (size_t i = 0; i < splits.size(); ++i) { -+ splits[i] /= split_sum; -+ } -+} -+ -+// Map the i_layer-th offloaded layer (of n_layers) to a device index, using the split points -+// produced by llama_model_splits_normalize(). -+int llama_model_splits_select_device(const std::vector & splits, int i_layer, int n_layers) { -+ const int idx = std::upper_bound(splits.begin(), splits.end(), float(i_layer)/n_layers) - splits.begin(); -+ -+ if (idx < 0 || (size_t) idx >= splits.size()) { -+ // Unreachable for well-formed split points, but indexing the device list with an -+ // out-of-range value gives std::out_of_range, whose libc++ what() is the bare string -+ // "vector" -- a message that names neither the container, nor the index, nor this -+ // function, and that the model-load error path prints verbatim. Say what went wrong, -+ // and print the split points, since a malformed one (NaN, unsorted, > 1) is the cause. -+ std::string s; -+ for (size_t i = 0; i < splits.size(); ++i) { -+ s += (i == 0 ? "" : ", ") + std::to_string(splits[i]); -+ } -+ throw std::runtime_error(format( -+ "%s: offloaded layer %d of %d mapped to device index %d, but there are only %zu " -+ "split point(s) [%s] -- the split points are malformed", -+ __func__, i_layer, n_layers, idx, splits.size(), s.c_str())); -+ } -+ -+ return idx; -+} -+ - bool llama_model_base::load_tensors(llama_model_loader & ml) { - const auto & split_mode = params.split_mode; - const bool use_mlock = params.load_mode == LLAMA_LOAD_MODE_MLOCK || params.load_mode == LLAMA_LOAD_MODE_MMAP_MLOCK; -@@ -1638,15 +1707,7 @@ bool llama_model_base::load_tensors(llama_model_loader & ml) { - std::copy(tensor_split, tensor_split + n_devices(), splits.begin()); - } - -- // sum and normalize the splits to get the split points -- float split_sum = 0.0f; -- for (size_t i = 0; i < n_devices(); ++i) { -- split_sum += splits[i]; -- splits[i] = split_sum; -- } -- for (size_t i = 0; i < n_devices(); ++i) { -- splits[i] /= split_sum; -- } -+ llama_model_splits_normalize(splits); - - const int i_gpu_start = std::max(n_layer_all + 1 - n_gpu_layers, 0); - const int act_gpu_layers = devices.empty() ? 0 : std::min(n_gpu_layers, n_layer_all + 1); -@@ -1656,7 +1717,7 @@ bool llama_model_base::load_tensors(llama_model_loader & ml) { - LLAMA_LOG_DEBUG("load_tensors: layer %3d assigned to device %s, is_swa = %d\n", il, ggml_backend_dev_name(cpu_dev), is_swa); - return {cpu_dev, &pimpl->cpu_buft_list}; - } -- const int layer_gpu = std::upper_bound(splits.begin(), splits.begin() + n_devices(), float(il - i_gpu_start)/act_gpu_layers) - splits.begin(); -+ const int layer_gpu = llama_model_splits_select_device(splits, il - i_gpu_start, act_gpu_layers); - auto * dev = devices.at(layer_gpu).dev; - LLAMA_LOG_DEBUG("load_tensors: layer %3d assigned to device %s, is_swa = %d\n", il, ggml_backend_dev_name(dev), is_swa); - return {dev, &pimpl->gpu_buft_list.at(dev)}; -diff --git a/src/llama-model.h b/src/llama-model.h -index dff84c574..3b1e98305 100644 ---- a/src/llama-model.h -+++ b/src/llama-model.h -@@ -919,6 +919,11 @@ struct llama_model_base : public llama_model { - - const char * llm_type_name(llm_type type); - -+// load_tensors() helpers, declared here so tests/test-model-split.cpp can drive them directly. -+// See their definitions in llama-model.cpp for what each one guarantees. -+void llama_model_splits_normalize(std::vector & splits); -+int llama_model_splits_select_device(const std::vector & splits, int i_layer, int n_layers); -+ - // convenience macro for loading local variables for load_tensors() in llama_model_base - // note: cast to int64_t since we will use these for the tensor dimensions - #define LLAMA_LOAD_LOCALS \ -diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt -index daccea063..6746023b1 100644 ---- a/tests/CMakeLists.txt -+++ b/tests/CMakeLists.txt -@@ -178,6 +178,7 @@ if (NOT WIN32 OR NOT BUILD_SHARED_LIBS) - llama_build_and_test(test-grammar-integration.cpp) - llama_build_and_test(test-llama-grammar.cpp LINK llama) - llama_build_and_test(test-batch-alloc.cpp LINK llama) -+ llama_build_and_test(test-model-split.cpp LINK llama) - llama_build_and_test(test-chat.cpp WORKING_DIRECTORY ${PROJECT_SOURCE_DIR}) - target_include_directories(test-chat PRIVATE ${PROJECT_SOURCE_DIR}/tools/server) - target_link_libraries(test-chat PRIVATE server-context) -diff --git a/tests/test-model-split.cpp b/tests/test-model-split.cpp -new file mode 100644 -index 000000000..35cc84666 ---- /dev/null -+++ b/tests/test-model-split.cpp -@@ -0,0 +1,131 @@ -+#include "testing.h" -+ -+#include "llama.h" -+ -+#include "../src/llama-model.h" -+ -+#include -+#include -+#include -+#include -+ -+// The lookup load_tensors() performs for every offloaded layer. A well-formed set of split points -+// must map every layer in [0, n_layers) to a valid device index; that invariant is what these -+// tests pin, because breaking it is what turns a model load into a std::out_of_range. -+static bool all_layers_map_into_range(testing & t, const std::vector & splits, int n_layers) { -+ for (int il = 0; il < n_layers; ++il) { -+ int idx = -1; -+ try { -+ idx = llama_model_splits_select_device(splits, il, n_layers); -+ } catch (const std::exception & e) { -+ t.assert_true(std::string("layer ") + std::to_string(il) + " threw: " + e.what(), false); -+ return false; -+ } -+ if (idx < 0 || (size_t) idx >= splits.size()) { -+ t.assert_true(std::string("layer ") + std::to_string(il) + " out of range", false); -+ return false; -+ } -+ } -+ return true; -+} -+ -+static void test_normalize_proportional(testing & t) { -+ std::vector splits = { 1.0f, 3.0f }; -+ llama_model_splits_normalize(splits); -+ -+ t.assert_true("first split point is 1/4", std::fabs(splits[0] - 0.25f) < 1e-6f); -+ t.assert_true("last split point is 1", std::fabs(splits[1] - 1.00f) < 1e-6f); -+ t.assert_true("all layers map into range", all_layers_map_into_range(t, splits, 32)); -+} -+ -+static void test_normalize_single_device(testing & t) { -+ std::vector splits = { 42.0f }; -+ llama_model_splits_normalize(splits); -+ -+ t.assert_true("the only split point is 1", std::fabs(splits[0] - 1.0f) < 1e-6f); -+ t.assert_true("all layers map into range", all_layers_map_into_range(t, splits, 32)); -+} -+ -+// The regression this file exists for. A device that reports zero free memory -- e.g. a Metal -+// device whose currentAllocatedSize has grown past its recommendedMaxWorkingSetSize -- makes the -+// sum of the weights zero. Normalizing by that sum used to divide by zero, putting NaN in every -+// split point; NaN compares false against everything, so std::upper_bound returned the end -+// iterator and load_tensors() indexed one past the last device. The resulting std::out_of_range -+// surfaced as "error loading model: vector", which names nothing. -+static void test_normalize_zero_sum(testing & t) { -+ for (size_t n_devices : { (size_t) 1, (size_t) 2, (size_t) 4 }) { -+ std::vector splits(n_devices, 0.0f); -+ llama_model_splits_normalize(splits); -+ -+ bool finite = true; -+ for (size_t i = 0; i < splits.size(); ++i) { -+ finite = finite && std::isfinite(splits[i]); -+ } -+ -+ t.assert_true("no NaN split point", finite); -+ t.assert_true("split points are even", std::fabs(splits.back() - 1.0f) < 1e-6f); -+ t.assert_true("all layers map into range", all_layers_map_into_range(t, splits, 32)); -+ } -+} -+ -+// The same degenerate sum, reached from the other direction: --tensor-split values are parsed with -+// std::stof and never range-checked, so `-ts 1,-1` cancels out and lands here on any backend, with -+// no GPU memory pressure involved at all. -+static void test_normalize_cancelling_tensor_split(testing & t) { -+ std::vector splits = { 1.0f, -1.0f }; -+ llama_model_splits_normalize(splits); -+ -+ bool finite = true; -+ for (size_t i = 0; i < splits.size(); ++i) { -+ finite = finite && std::isfinite(splits[i]); -+ } -+ -+ t.assert_true("no NaN or inf split point", finite); -+ t.assert_true("all layers map into range", all_layers_map_into_range(t, splits, 32)); -+} -+ -+// The diagnostic half: a malformed set of split points must name itself rather than reaching the -+// device list and coming back out as libc++'s bare "vector". -+static void test_select_device_reports_malformed_splits(testing & t) { -+ const std::vector nan_splits = { std::nanf(""), std::nanf("") }; -+ -+ bool threw = false; -+ std::string what; -+ try { -+ llama_model_splits_select_device(nan_splits, 0, 32); -+ } catch (const std::exception & e) { -+ threw = true; -+ what = e.what(); -+ } -+ -+ t.assert_true("malformed split points throw", threw); -+ t.assert_true("message names the function", what.find("llama_model_splits_select_device") != std::string::npos); -+ t.assert_true("message names the layer", what.find("layer 0 of 32") != std::string::npos); -+ t.assert_true("message names the device index", what.find("device index 2") != std::string::npos); -+ t.assert_true("message prints the split points", what.find("nan") != std::string::npos); -+ t.assert_true("message is not just \"vector\"", what != "vector"); -+} -+ -+int main(int argc, char ** argv) { -+ testing t; -+ -+ const char * verbose = getenv("LLAMA_TEST_VERBOSE"); -+ if (verbose) { -+ t.verbose = std::string(verbose) == "1"; -+ } -+ if (!t.verbose) { -+ llama_log_set([](ggml_log_level, const char *, void *) {}, nullptr); -+ } -+ -+ if (argc > 1) { -+ t.set_filter(argv[1]); -+ } -+ -+ t.test("normalize_proportional", test_normalize_proportional); -+ t.test("normalize_single_device", test_normalize_single_device); -+ t.test("normalize_zero_sum", test_normalize_zero_sum); -+ t.test("normalize_cancelling_ts", test_normalize_cancelling_tensor_split); -+ t.test("select_device_malformed", test_select_device_reports_malformed_splits); -+ -+ return t.summary(); -+} diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index 4a816003..c7e99072 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -9,13 +9,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b11617"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b11632"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b11617-"} — call + * plus the resolved upstream commit, e.g. {@code "b11632-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -23,14 +23,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b11617"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b11632"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b11617"; + public static final String LLAMA_CPP_VERSION = "b11632"; // Constants holder — not instantiable. private LlamaCppVersion() {} diff --git a/llama/src/test/cpp/test_model_split.cpp b/llama/src/test/cpp/test_model_split.cpp deleted file mode 100644 index 82194c51..00000000 --- a/llama/src/test/cpp/test_model_split.cpp +++ /dev/null @@ -1,126 +0,0 @@ -// SPDX-FileCopyrightText: 2026 Bernard Ladenthin -// -// SPDX-License-Identifier: MIT -// -// Runnable guard for patches/0012 (the layer-split fix in llama.cpp's src/llama-model.cpp). -// -// The patch also ships an upstream test (tests/test-model-split.cpp), but a FetchContent -// subproject builds with LLAMA_BUILD_TESTS=OFF, so that one is applied-but-never-compiled here. -// This file drives the same two functions from jllama_test, which every platform runs in CI -- -// so a llama.cpp bump that drops the patch reds "C++ Tests" everywhere instead of surfacing as -// one red macOS Java job with the message "error loading model: vector". -// -// Note that the failure it guards against is NOT reproducible on this side: it needs a GPU -// backend that reports zero free memory, and without one `devices` is empty and the mapping is -// never reached. What is testable -- and is what actually broke -- is the arithmetic itself. -// -// Upstream fixed the defect itself on 2026-10-14 (ggml-org/llama.cpp#29677, commit df18bee11, -// https://github.com/ggml-org/llama.cpp/commit/df18bee11e96557b07aa7b12db0976f3c7ac9165): the same -// even-split fallback, inline in load_tensors(), and no test. The commit is in no tag up to b11618 -// (b11617 is the pin at the time of writing), so the patch and this guard still stand. At the first -// bump to a tag that carries it, patches/0012 is dropped, and with it the two functions this file -// links -- see TODO.md, "Drop patch 0012 at the first bump that carries #29677", for what replaces -// this guard. - -#include "llama-model.h" - -#include - -#include -#include -#include - -namespace { - -// The lookup load_tensors() performs for every offloaded layer. -void expect_every_layer_maps_into_range(const std::vector &splits, int n_layers) { - for (int il = 0; il < n_layers; ++il) { - int idx = -1; - ASSERT_NO_THROW(idx = llama_model_splits_select_device(splits, il, n_layers)) << "layer " << il; - EXPECT_GE(idx, 0) << "layer " << il; - EXPECT_LT((size_t)idx, splits.size()) << "layer " << il; - } -} - -} // namespace - -TEST(LlamaModelSplits, NormalizeIsProportionalToTheWeights) { - std::vector splits = {1.0f, 3.0f}; - llama_model_splits_normalize(splits); - - EXPECT_NEAR(splits[0], 0.25f, 1e-6f); - EXPECT_NEAR(splits[1], 1.00f, 1e-6f); -} - -TEST(LlamaModelSplits, NormalizeSingleDeviceTakesEverything) { - std::vector splits = {42.0f}; - llama_model_splits_normalize(splits); - - ASSERT_EQ(splits.size(), 1u); - EXPECT_NEAR(splits[0], 1.0f, 1e-6f); -} - -// The regression. A device reporting zero free memory -- e.g. a Metal device whose -// currentAllocatedSize has grown past its recommendedMaxWorkingSetSize -- makes the sum of the -// weights zero. Dividing by it put a NaN in every split point. -TEST(LlamaModelSplits, ZeroSumDoesNotProduceNaNSplitPoints) { - for (size_t n_devices : {(size_t)1, (size_t)2, (size_t)4}) { - std::vector splits(n_devices, 0.0f); - llama_model_splits_normalize(splits); - - ASSERT_EQ(splits.size(), n_devices); - for (size_t i = 0; i < splits.size(); ++i) { - EXPECT_TRUE(std::isfinite(splits[i])) << "n_devices=" << n_devices << " i=" << i; - } - EXPECT_NEAR(splits.back(), 1.0f, 1e-6f) << "n_devices=" << n_devices; - } -} - -// NaN compares false against everything, so std::upper_bound returned the end iterator and -// load_tensors() indexed one past the last device -- an std::out_of_range whose libc++ what() is -// the bare string "vector". This is the assertion that would have failed before the fix. -TEST(LlamaModelSplits, ZeroSumStillMapsEveryLayerToARealDevice) { - for (size_t n_devices : {(size_t)1, (size_t)2, (size_t)4}) { - std::vector splits(n_devices, 0.0f); - llama_model_splits_normalize(splits); - expect_every_layer_maps_into_range(splits, 32); - } -} - -// The same degenerate sum reached from the other direction, and the reason this is not a -// Metal-only defect: --tensor-split values are parsed with std::stof and never range-checked, so -// `-ts 1,-1` cancels out and lands on the very same line, on any backend, with no memory pressure. -// Unfixed this yields [inf, -nan] and every layer maps one past the last device. -TEST(LlamaModelSplits, CancellingTensorSplitStillMapsEveryLayerToARealDevice) { - std::vector splits = {1.0f, -1.0f}; - llama_model_splits_normalize(splits); - - for (size_t i = 0; i < splits.size(); ++i) { - EXPECT_TRUE(std::isfinite(splits[i])) << "i=" << i; - } - expect_every_layer_maps_into_range(splits, 32); -} - -TEST(LlamaModelSplits, ProportionalSplitsMapEveryLayerToARealDevice) { - std::vector splits = {1.0f, 3.0f}; - llama_model_splits_normalize(splits); - expect_every_layer_maps_into_range(splits, 32); -} - -// The diagnostic half: malformed split points must name themselves rather than reaching the -// device list and coming back out as libc++'s content-free "vector". -TEST(LlamaModelSplits, SelectDeviceNamesMalformedSplitPoints) { - const std::vector nan_splits = {std::nanf(""), std::nanf("")}; - - try { - llama_model_splits_select_device(nan_splits, 0, 32); - FAIL() << "expected malformed split points to throw"; - } catch (const std::exception &e) { - const std::string what = e.what(); - EXPECT_NE(what, "vector"); - EXPECT_NE(what.find("llama_model_splits_select_device"), std::string::npos) << what; - EXPECT_NE(what.find("layer 0 of 32"), std::string::npos) << what; - EXPECT_NE(what.find("device index 2"), std::string::npos) << what; - EXPECT_NE(what.find("nan"), std::string::npos) << what; - } -}