diff --git a/CLAUDE.md b/CLAUDE.md index 18280df70..607f216e8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b11574** +Current llama.cpp pinned version: **b11612** ## Natives jars: library jars and GPU module jars (`.github/natives.csv`) @@ -178,7 +178,7 @@ measurements behind the Windows step are in `docs/handover/local-agent-report-b1 `add_subdirectory(ggml)` when a `ggml` target exists; a module-only build never adds llama.cpp). llama, mtmd and llama-common stay static in `libjllama.so`. **Every ggml default llama.cpp's own CMakeLists changes before its `add_subdirectory(ggml)` is repeated** -- `GGML_LLAMAFILE_DEFAULT` - and `GGML_CUDA_GRAPHS_DEFAULT` at b11574 -- because adding ggml first skips that block; re-check with + and `GGML_CUDA_GRAPHS_DEFAULT` at b11612 -- because adding ggml first skips that block; re-check with `grep -n _DEFAULT CMakeLists.txt` at the new tag on every bump. The second one was missing once: ggml's own default for CUDA graphs is OFF, and a CUDA module built without it lost two thirds of its token generation (265.6 -> 88.8 tg128 on an RTX 3070, prompt processing -20%; Vulkan, which @@ -1106,7 +1106,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network for the asset build; the embed step is plain cmake -P -git clone --depth 1 --branch b11574 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b11612 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build ) mkdir -p webui-generated /tmp/ui-gen cmake -DUI_SOURCE_DIR=/tmp/lc/tools/ui -DUI_BINARY_DIR=/tmp/ui-gen \ @@ -1146,7 +1146,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b11574`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b11612`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1370,18 +1370,18 @@ Current patches: | Patch | Fixes | |-------|-------| -| `0001-win32-arg-parse-embed-guard.patch` | Windows JNI regression from llama.cpp **#24779** (introduced b9739): on Windows `common_params_parse` re-derived argv from the **process** command line (`GetCommandLineW`) and adopted it, so an embedded/JNI caller (`java.exe`) lost its `--model …` args → "Failed to parse model parameters". b9789 narrowed the unconditional override to a **count-guard** (`if (static_cast(utf8.buf.size()) == argc) { argv = utf8.ptrs.data(); }`), but that is exactly the variant the project already found breaks its Windows server-integration tests (when the embedded argv length coincides with `java.exe`'s). The patch carries the **complete upstream change** (so it can be submitted to llama.cpp verbatim and then dropped here): **(1)** `common_params_parse` parses **exactly the argv it is given** (no `GetCommandLineW` magic) and a new `common_params_parse_main()` wrapper holds the UTF-8 recovery for the standalone tools' `main()` (`common/arg.{cpp,h}`); **(2)** the **~34 standalone `main()` call sites** (every `common_params_parse(argc, argv, …)` across `tools/*`, `examples/*` and the `tests/*` programs) flip to `common_params_parse_main()`; **(3)** a `tests/test-arg-parser.cpp` regression case pins that `common_params_parse` honors a caller-supplied argv. The embedded caller (`jllama.cpp`) keeps calling `common_params_parse` and is never overridden. **Our subproject build compiles only the `arg.{cpp,h}` core** — `LLAMA_BUILD_TOOLS`/`LLAMA_BUILD_TESTS` are OFF for a FetchContent subproject — so the flips + test are applied-but-not-compiled here; they were validated via a one-off `-DLLAMA_BUILD_TOOLS=ON -DLLAMA_BUILD_TESTS=ON` build (the new test compiles and its asserts pass; `test-arg-parser`'s only red there is the live `ggml.ai` download check, which is sandbox-network, not the patch). Because it spans **35 files** it must be refreshed on every llama.cpp bump (the applier fails loud). **Refreshed at the b10679 bump:** upstream rewrote `tests/test-save-load-state.cpp`'s `main()` to take a `--models DIR` option, which it strips itself into a `filtered_argv` before calling `common_params_parse(fargc, filtered_argv.data(), …)`. That call site therefore stopped qualifying for the `_main()` flip — by this patch's own rule a caller that builds its own argv must use `common_params_parse` directly, so its argv is kept — and the hunk was **dropped** rather than refreshed (37 → 36 files). Caveat for whoever submits this upstream: that `main()` now filters a possibly-mojibake Windows argv *before* any UTF-8 recovery, so the fully correct upstream form there is recover-then-filter, not a one-line flip. It is out of scope for the downstream carry because `LLAMA_BUILD_TESTS` is OFF here, so the file is never compiled. **Refreshed the same way at the b11236 bump:** upstream #29426 rewrote `tests/test-recurrent-state-rollback.cpp`'s `main()` into the identical `--models DIR` / `filtered_argv` shape, so that hunk was dropped too (36 → 35 files); the same recover-then-filter caveat applies to it. **Still required at b10679, verified rather than assumed:** `common_params_parse` in pristine `b10679:common/arg.cpp` still carries the `#ifdef _WIN32` count-guarded `argv = utf8.ptrs.data()` override, and `common_params_parse_main` appears nowhere in `b10679:common/arg.h` — upstream has not adopted the fix. The upstream-facing write-up, including a standalone reproducer that makes llama.cpp's own `test-arg-parser` fail on unmodified `master`, lives in [docs/upstream-investigation-win32-argv-substitution.md](docs/upstream-investigation-win32-argv-substitution.md). **Reported upstream as [ggml-org/llama.cpp#26416](https://github.com/ggml-org/llama.cpp/issues/26416)** (2026-08-01, label `bug-unconfirmed`, first bad commit `508a475`); the issue asks which of the two directions the maintainers prefer before a PR is opened, so this patch stays downstream until they answer. | +| `0001-win32-arg-parse-embed-guard.patch` | Windows JNI regression from llama.cpp **#24779** (introduced b9739): on Windows `common_params_parse` re-derived argv from the **process** command line (`GetCommandLineW`) and adopted it, so an embedded/JNI caller (`java.exe`) lost its `--model …` args → "Failed to parse model parameters". b9789 narrowed the unconditional override to a **count-guard** (`if (static_cast(utf8.buf.size()) == argc) { argv = utf8.ptrs.data(); }`), but that is exactly the variant the project already found breaks its Windows server-integration tests (when the embedded argv length coincides with `java.exe`'s). The patch carries the **complete upstream change** (so it can be submitted to llama.cpp verbatim and then dropped here): **(1)** `common_params_parse` parses **exactly the argv it is given** (no `GetCommandLineW` magic) and a new `common_params_parse_main()` wrapper holds the UTF-8 recovery for the standalone tools' `main()` (`common/arg.{cpp,h}`); **(2)** the **~34 standalone `main()` call sites** (every `common_params_parse(argc, argv, …)` across `tools/*`, `examples/*` and the `tests/*` programs) flip to `common_params_parse_main()`; **(3)** a `tests/test-arg-parser.cpp` regression case pins that `common_params_parse` honors a caller-supplied argv. The embedded caller (`jllama.cpp`) keeps calling `common_params_parse` and is never overridden. **Our subproject build compiles only the `arg.{cpp,h}` core** — `LLAMA_BUILD_TOOLS`/`LLAMA_BUILD_TESTS` are OFF for a FetchContent subproject — so the flips + test are applied-but-not-compiled here; they were validated via a one-off `-DLLAMA_BUILD_TOOLS=ON -DLLAMA_BUILD_TESTS=ON` build (the new test compiles and its asserts pass; `test-arg-parser`'s only red there is the live `ggml.ai` download check, which is sandbox-network, not the patch). Because it spans **35 files** it must be refreshed on every llama.cpp bump (the applier fails loud). **Refreshed at the b10679 bump:** upstream rewrote `tests/test-save-load-state.cpp`'s `main()` to take a `--models DIR` option, which it strips itself into a `filtered_argv` before calling `common_params_parse(fargc, filtered_argv.data(), …)`. That call site therefore stopped qualifying for the `_main()` flip — by this patch's own rule a caller that builds its own argv must use `common_params_parse` directly, so its argv is kept — and the hunk was **dropped** rather than refreshed (37 → 36 files). Caveat for whoever submits this upstream: that `main()` now filters a possibly-mojibake Windows argv *before* any UTF-8 recovery, so the fully correct upstream form there is recover-then-filter, not a one-line flip. It is out of scope for the downstream carry because `LLAMA_BUILD_TESTS` is OFF here, so the file is never compiled. **Refreshed the same way at the b11236 bump:** upstream #29426 rewrote `tests/test-recurrent-state-rollback.cpp`'s `main()` into the identical `--models DIR` / `filtered_argv` shape, so that hunk was dropped too (36 → 35 files); the same recover-then-filter caveat applies to it. **Still required at b10679, verified rather than assumed:** `common_params_parse` in pristine `b10679:common/arg.cpp` still carries the `#ifdef _WIN32` count-guarded `argv = utf8.ptrs.data()` override, and `common_params_parse_main` appears nowhere in `b10679:common/arg.h` — upstream has not adopted the fix. The upstream-facing write-up, including a standalone reproducer that makes llama.cpp's own `test-arg-parser` fail on unmodified `master`, lives in [docs/upstream-investigation-win32-argv-substitution.md](docs/upstream-investigation-win32-argv-substitution.md). **Reported upstream as [ggml-org/llama.cpp#26416](https://github.com/ggml-org/llama.cpp/issues/26416)** (2026-08-01, label `bug-unconfirmed`, first bad commit `508a475`); the issue asks which of the two directions the maintainers prefer before a PR is opened, so this patch stays downstream until they answer. **Refreshed at the b11612 bump:** #30023 (`a17e94d22`, "tests : consolidate context state tests into test-llama-context") deleted `tests/test-state-restore-fragmented.cpp`, so its hunk was dropped (36 → 35 `diff --git` headers); the consolidated `tests/test-llama-context.cpp` has the `--models DIR` / `filtered_argv` `main()` shape, so by this patch's own rule it keeps calling `common_params_parse` directly and gets no flip. Still required, verified at pristine b11612: `common/arg.cpp:1288` carries the count-guarded `argv = utf8.ptrs.data()` override and `common/arg.h` has no `common_params_parse_main`. | | `0002-server-preserve-caller-load-progress-callback.patch` | Load-progress-callback regression introduced in llama.cpp **b9789**: `server_context::load_model` (`tools/server/server-context.cpp`) now **unconditionally** installs the server's own load-progress reporter on `params_base.load_progress_callback` immediately before `common_init_from_params`, clobbering any callback the embedding caller already set. libjllama's `LoadProgressCallback` feature wires `common_params.load_progress_callback` to a JNI trampoline *before* calling `load_model`, so the bump silently killed it — `LoadProgressCallbackTest` saw zero progress updates and the abort-on-`false` path never threw. The patch guards the assignment with `if (params_base.load_progress_callback == nullptr)`, so the server installs its own reporter **only when the caller hasn't** — a caller-supplied callback survives and fires during load. Standalone `llama-server` (no caller callback, so the field is null) is unaffected. Same JNI-vs-standalone divergence class as `0001`. **The guard is `== nullptr || == load_progress_callback`, and the second disjunct must never be dropped:** `load_progress_text` is a **local** of `load_model()`, and upstream re-assigns both fields on every call so the `user_data` always points at the current frame. `load_model()` runs a **second** time when resuming from the sleeping state (`--sleep-idle-seconds`), and by then `params_base` holds *our own* callback from the first load — a bare nullptr check skips the re-assignment and leaves `user_data` pointing into a **dead stack frame**, which segfaults inside `load_progress_callback()` on the first request after an idle window. That was a latent defect in this patch from the day it was written; only a second `load_model()` can reach it, and nothing exercised sleep until `IdleSleepWakeIntegrationTest` was added. | | `0003-pr22393-server-add-slot-prompt-similarity-getter-setter.patch` | **Upstream-PR carry** of [ggml-org/llama.cpp#22393](https://github.com/ggml-org/llama.cpp/pull/22393) ("server : add slot_prompt_similarity getter/setter"). Purely additive: adds `server_context::get_slot_prompt_similarity()` / `set_slot_prompt_similarity(float)` (`tools/server/server-context.{cpp,h}`) so an embedding/JNI caller can query and tune the slot-selection threshold at runtime without reloading the model. Verbatim copy of the PR, which **upstream closed without merging** (rejected as exposing unsafe internal state — see the patch header). Carried permanently; it will not be droppable via a version bump. | | `0007-server-attach-http-frontend.patch` | **Adds `llama_server_attach(argc, argv, server_context&)`** so the `NativeServer` *attach mode* can serve an **already-loaded `LlamaModel`** over the upstream HTTP frontend — no second model load, no `start_loop()`; the LlamaModel's worker keeps driving the shared `server_context` and the HTTP routes post tasks to its queue (the queue is the synchronization point). Mechanically: (1) extracts the **pure core route table** (`health` … `slots`) out of `llama_server()` into `static void llama_server_register_common_routes(ctx_http, routes)` (shared, so the two entry points cannot drift on the core endpoint set). **Scope note (narrowed at the b10154 bump):** the helper deliberately carries **only** the stable, state-independent route table — **not** the resumable-streaming routes (their handlers differ between router / non-router), the GCP-compat shim, or the experimental **CORS-proxy / MCP-server / built-in-tools** wiring. b10154 (upstream MCP-server support) moved the streaming routes into the middle of that block and coupled tools/CORS to a per-call `server_mcp mcp_mgr` lifecycle, so the earlier contiguous "route-table + CORS-proxy + tools" extraction is no longer possible; `llama_server()` keeps all of that inline, **byte-identical to upstream b10154** (only the route-table block is factored out). (2) adds `llama_server_attach`, which parses only the HTTP-side argv via `common_params_parse`, starts the stream-session GC + `server_http_context`, registers the common route table, the **non-router** resumable-streaming handlers (upstream b10154 paths `/v1/stream` GET/DEL + `/v1/streams/lookup` POST), the GCP-compat shim, and **403 "disabled" stubs for `/cors-proxy` + `/tools`** (attach mode does not wire the experimental CORS-proxy / MCP / built-in-tools host — those belong to a full `llama-server`, not an embedded model), marks ready immediately (model already loaded), and blocks on the HTTP thread until `llama_server_request_shutdown()` — never calling `common_init()`, backend init, `ctx_server.terminate()` or `llama_backend_free()` (the embedding caller owns those). Applies after `0001`+`0006` (same file); closes the "NativeServer — reuse an already-loaded LlamaModel" TODO. Upstream-submittable ("server: let embedding callers attach the HTTP frontend to an existing server_context"). **Refreshed at the b10519 bump:** upstream #26347 dropped the API key from the `/models` + `/v1/models` public-endpoint set and deleted the two trailing `// public endpoint (no API key check)` comments on those route registrations. Those two lines sit inside this patch's route-table removal block, so `git apply` failed ("patch does not apply", `server.cpp:258`) at **every** tag from b10519 on; the fix was to drop the now-wrong comment from all four affected lines (2 on the `-` side, 2 in the extracted helper on the `+` side), keeping the helper byte-identical to the block it replaces. **This is the invariant to re-check on every bump:** the `+` side of `llama_server_register_common_routes()` must stay a verbatim copy of the route table it factors out of `llama_server()`. **Refreshed at the b11104 bump** (upstream #28690, multi-address `--host`): `server_http_context` lost its single `thread` and `listening_address` members in favour of `join()` and a `listening_addresses` vector, one listener thread per bound address. The patch still *applied* cleanly there — only its own `+` lines named the removed members — so the applier could not see it; `llama_server_attach` now logs every address and blocks in `ctx_http.join()`, exactly as upstream's `llama_server()` does. **Refreshed at the b11361 bump** (#29818 adds `POST /v1/systemone` to the route table: one line, added on both sides so the helper stays verbatim) **and at b11401** (#29895 adds a `server_child &` overload of `llama_server()` beside the declarations and a `server_child child;` at the top of the argv entry point, the context of `0007`'s first two hunks; only context moved, the `+`/`-` lines are unchanged). **Changed in 5.2.x, not a refresh: `llama_server_attach(argc, argv, server_context &, server_routes &)` serves the caller's `server_routes`** (`jllama_context::routes`, the one the model load builds) **instead of constructing one on its own frame.** `server_routes`' constructor registers a sleeping-state callback on the model's task queue, `server_queue::on_sleeping_state` only appends, so the frame-local object left a callback into freed memory behind when the attach returned, and the model's next idle sleep (`--sleep-idle-seconds`) ran it -- undefined behaviour, found at the b11361 bump and pinned by `NativeServerAttachSleepIntegrationTest` (model-backed, CI). Consequence: the route handlers read the **model's** params, so the endpoint toggles (`--metrics`, `--props`, `--slots`), `--slot-save-path`, the slot count and the sampling defaults are `ModelParameters` now (`enableMetricsEndpoint()` & co., added for that), and the attach argv configures the HTTP side only. | | `0008-server-models-worker-cmd-override.patch` | **Makes router mode usable in-JVM.** The router (`server-models.cpp`) spawns each model worker by re-executing its own binary (`get_server_exec_path()` = `/proc/self/exe` & friends) — inside a JVM that binary is `java`, not a llama-server, so embedded router workers could never start. The patch adds env `LLAMA_SERVER_WORKER_CMD` (whitespace-split; read in `server_model_meta::update_args`) which replaces only the leading binary-path token of the rendered worker args, letting an embedding host relaunch workers through its own bootstrap — e.g. `java -cp app.jar net.ladenthin.llama.server.NativeServer` (each worker is then a fresh JVM running the classic single-model `NativeServer`). Exposed in Java as `NativeServer.setWorkerCommand(String...)` (JNI `setenv`); exercised by `RouterModeIntegrationTest` (Linux CI). Upstream-submittable (also useful for containerized/wrapped deployments). **Refreshed at the b11429 bump:** #29987 (untagged `4d60b4d08`, the commit before b11429) moved `server_model_meta::update_args` and changed the function after it (`update_caps(const common_params &)`), the hunk's trailing context; the `+` lines are unchanged. Since b11401 (#29895) a worker is a router *child* whose stdout carries the state commands, which is why `LlamaLoader` prints its two diagnostic lines to stderr (a JVM worker's early `System.out` line tripped the router until 2026-10-09, `e9758054`). | | `0006-server-embed-native-server-jni.patch` | **Makes `server.cpp`'s `llama_server` embeddable in the JVM** so the `NativeServer` JNI bridge can run the full upstream HTTP server (WebUI included) inside `libjllama` — see "Two server modes" below. b9870 already exposes `int llama_server(int, char**)` (non-static; no `main` in the file), so the patch only adds embedded-mode support: (1) a `g_llama_server_embedded` flag + `llama_server_set_embedded()` / `llama_server_request_shutdown()` (declared in the committed `src/main/cpp/native_server_bridge.h`); (2) skips installing the process-wide SIGINT/SIGTERM handlers when embedded (they would hijack the JVM's); (3) in embedded mode parses the **forwarded** argv via `common_params_parse` instead of `common_params_parse_main` (whose `GetCommandLineW` recovery would pick up `java.exe`'s command line — the same Windows class of bug `0001` fixes). `llama_server_request_shutdown()` mirrors the SIGTERM path (invokes the installed `shutdown_handler` → `ctx_server.terminate()` unblocks `start_loop()`), giving JNI an out-of-band stop since `ctx_server` is loop-local. **The handler is guarded, and both guards are load-bearing:** upstream's `shutdown_handler` lambdas capture `llama_server()`'s locals (`ctx_http`, `models_routes`, `mcp_mgr`, `ctx_server`) by reference and are never cleared, while `native_server.cpp` signals *repeatedly* until the worker reports it finished (a stop issued before the handler is installed, or before `start_loop()` begins, would otherwise be lost). A request landing after `llama_server()` returned therefore ran the lambda over destroyed objects — a `SIGSEGV` in `server_http_context::stop()` in `RouterModeIntegrationTest.tearDown` (run 36485159455), with router mode's `clean_up()` (unloading workers) making the window wide. The patch (1) takes `g_shutdown_handler_mutex` for every read, write and invocation of the handler (the caller is on another thread) and (2) declares a `llama_server_shutdown_handler_reset` **after** the captured locals, so the handler is cleared before any of them is destroyed; because invocation holds the same lock, that reset waits for an invocation in progress. `0007`'s `llama_server_attach` carries the same guard. Runnable guard: `src/test/cpp/test_native_server_shutdown.cpp`. Applies **after `0001`** (which flips this call site to `common_params_parse_main`), so its context is the post-`0001` tree; regenerate against `0001`+source on a bump. Only touches `tools/server/server.cpp`. | -| `0012-model-guard-zero-split-sum-and-name-the-device-index.patch` | **A GPU that reports zero free memory makes every model load fail with the unactionable `error loading model: vector`.** `llama_model_base::load_tensors` (`src/llama-model.cpp`) weights the per-device layer split by `ggml_backend_dev_memory()`'s `free`, then normalises: `splits[i] /= split_sum`. With a single device reporting `free == 0` that is `0/0` → **NaN** in every split point; NaN compares false against everything, so the `std::upper_bound` below returns the end iterator, `layer_gpu == n_devices()`, and `devices.at(layer_gpu)` throws `std::out_of_range` — whose libc++ `what()` is the bare string `"vector"`, which `llama.cpp`'s `catch (const std::exception &)` prints verbatim. Upstream's `free == 0 && total == 0` host-memory fallback does **not** fire, because `total` is `recommendedMaxWorkingSetSize` and is non-zero. **Reachable since b10618..b10797**: upstream `8c0b9cd04` ("metal : fix memory query under low-memory conditions", [#27701](https://github.com/ggml-org/llama.cpp/pull/27701)) changed `ggml-metal-device.m` to `*free = *total > cur ? *total - cur : 0`; before that clamp an over-committed device (`currentAllocatedSize > recommendedMaxWorkingSetSize`) *underflowed* to a huge `size_t`, which normalised fine, so the same precondition was harmless. That is why the `Java Tests macOS …` jobs went red at the b10792→b10797 step while every Linux/Windows job stayed green — **and why only a GPU build can fail this way at all**: `act_gpu_layers` is `devices.empty() ? 0 : …`, so with no GPU backend `devices` is empty, every layer returns early on `cpu_dev`, and the `.at()` line is unreachable. **Shape:** the two blocks are lifted out of `load_tensors` into free functions declared in `src/llama-model.h`, purely so they can be driven by a test — the failing state needs a real over-committed GPU and cannot be arranged through any public API. `llama_model_splits_normalize()` carries **the fix**: on `split_sum == 0` it `LLAMA_LOG_WARN`s and falls back to an even split (`splits[i] = float(i+1)/splits.size()`), the only neutral choice when no device can be preferred and exactly right for a single device. `llama_model_splits_select_device()` carries **the diagnostic**: it bounds-checks the index and throws a `std::runtime_error` naming the function, the offloaded layer, the device index, the split-point count **and the split points themselves** — with NaN splits that message prints `nan` and names the cause outright, which is precisely what was missing when this had to be diagnosed by reading source. **A second, backend-independent trigger reaches the same line**, found while writing this up and verified against the unfixed library: `--tensor-split` values are parsed with `std::stof` and never range-checked (`common/arg.cpp`), so `-ts 1,-1` cancels out, `split_sum` is 0 again, the split points become `[inf, -nan]`, and every layer maps one past the last device — on CUDA, Vulkan or ROCm just as much as on Metal, with no memory pressure involved. That is what makes this an ordinary upstream defect rather than a Metal edge case, and the warning names both causes rather than only the memory one. Also adds upstream `tests/test-model-split.cpp` (5 cases in upstream's `testing.h` style) + its `llama_build_and_test` registration. Touches `src/llama-model.{cpp,h}`, `tests/test-model-split.cpp` and `tests/CMakeLists.txt` — **none** of which any other patch touches, so it is independent of all of them. **Refreshed at the b11570 bump** (#30299 links each upstream test against only the libraries it uses: `llama_build_and_test` takes `LINK `, and the registration this patch adds reads `llama_build_and_test(test-model-split.cpp LINK llama)` now -- the carried test includes `../src/llama-model.h` and needs `llama` only, like the other internal-function tests in that block; the `src/llama-model.{cpp,h}` and `tests/test-model-split.cpp` hunks are unchanged, only their offsets moved). Upstream-submittable ("model: fall back to an even split when no device reports free memory"); **not yet filed upstream**. **Runnable guard: `src/test/cpp/test_model_split.cpp`** — a FetchContent subproject builds with `LLAMA_BUILD_TESTS=OFF`, so the upstream test above is applied-but-never-compiled here (same as `0001`'s test). That file drives the same two functions from `jllama_test`, which runs on **every** platform in `C++ Tests`, so a bump that drops this patch fails the build at link time everywhere instead of surfacing as one red macOS Java job. **Verification limit — read before assuming this can be dropped:** the *failing path* still cannot be reached without a GPU backend, so the guard pins the arithmetic (what actually broke), not the end-to-end load; the end-to-end proof is the macOS CI job. On a bump, re-check whether upstream added its own `split_sum == 0` guard (grep `split_sum` in `src/llama-model.cpp`) and **drop this patch rather than refreshing it** if they did — the fail-loud applier detects "does not apply", never "upstream already fixed this". | -| `0014-common-log-callback-sink.patch` | **Gives `common_log` a callback sink: `common_log_set_callback(log, cb, user_data)` (`common/log.{h,cpp}`).** This is what `LlamaModel.setLogger` hooks. Before it, the Java logger was a `llama_log_set()` callback, which has two holes, both found while chasing `slot print_timing` lines interleaving with the Atmosphere agent's streamed answer: **(a)** every model load runs `common_init()`, which re-points `llama_log_set()` at `common_log_default_callback` (`common.cpp:394`), so `setLogger(…)` *before* `new LlamaModel(…)` silently lost the callback; **(b)** the server's own `SRV_*`/`SLT_*` macros are `LOG_INF` and write straight into `common_log`, which `llama_log_set()` never carried, so the per-request `slot …`/`srv …` lines could not be routed to Java at all (the reason `LlamaModelTest#testLogText/JSON` sat `@Disabled` for years). `common_log` upstream offers file, colors, prefix, timestamps, verbosity and JSONL but no hook. The patch adds one: while a callback is set, the worker thread hands every entry to it **instead of** printing to stdout/stderr (a `--log-file` still receives them); the callback gets the bare formatted message (no prefix/timestamp/colors) with the `ggml_log_callback` signature; swapping the callback pauses the worker first, so queued entries reach the *previous* sink (which is what makes `setLogger(format, null)` a synchronous drain). With the sink, `common_init()`'s `llama_log_set()` reset is harmless — it points at the default callback that feeds `common_log`, i.e. exactly the path into the sink — so the ordering problem (a) disappears without any re-install logic, and the `srv`/`slot` lines (b) arrive because they are `common_log` entries. `jllama.cpp`'s `setLogger` therefore sets `common_log_set_callback(common_log_main(), trampoline)` **plus** `llama_log_set(common_log_default_callback)` (so llama/ggml lines feed `common_log` even before the first load). Two consequences to know: the callback runs on `common_log`'s **worker thread**, a plain `std::thread` llama.cpp re-creates on every pause/resume and never attaches to the JVM — the trampoline attaches it **once per worker thread** as a daemon, named `jllama-log-worker`, and a `thread_local` guard detaches it when the thread exits (`get_jni_env_attaching`; `common_log::pause()` joins the worker, a normal exit on which TLS destructors run, and `JNI_OnUnload` pauses it before anything else). Until 5.2.x it attached and detached per line, which created a `java.lang.Thread` object per log line (13 lines of a failed load, 13 `Thread` objects -- `LlamaLoggerTest` pins the single attach now). Each call runs inside a `PushLocalFrame`/`PopLocalFrame` pair, because a thread that stays attached and has no Java frame would otherwise accumulate local references, `setLogger` must call `common_log_set_callback` **outside** `g_log_mutex`, because the pause joins the worker, which needs that mutex to read the callback, and `setLogger` callers are serialized by a **separate** `g_set_logger_mutex`: two unserialized swaps race on the worker's `std::thread` (one joins it while the other assigns a fresh thread over the still-joinable object = `std::terminate`, the whole JVM), which `LlamaLoggerTest#concurrentSetLoggerCallsDoNotRaceOnTheLogWorker` reproduced before the mutex existed. Two caveats the Javadoc carries: a caller must not hold a lock the *previous* callback needs (the drain runs it on the worker while the caller waits), and the verbosity threshold is process-wide and reset by **every** load (`common_params_parse` ends with `common_log_set_verbosity_thold(params.verbosity)`, default 3), so a load without `-lv` puts it back to 3 — a review assumed the opposite, and `LlamaLoggerTest#verbosityThresholdIsProcessWideAndEveryLoadSetsIt` now pins the measured behaviour. And the verbosity threshold applies *before* the sink: at the default (`3`) the Java logger sees errors, warnings and the server's INFO lines, while llama/ggml INFO lines (`common_log_get_verbosity` maps them to TRACE = 4) arrive only from `setLogVerbosity(4)` on — the same filtering the console gets, and a behaviour change for consumers who captured the unfiltered `llama_log_set()` stream before. **Runnable guards:** `src/test/cpp/test_common_log_callback.cpp` (6 tests over a private `common_log_init()` instance: delivery, bare text under prefix+timestamps, clear, swap-drains-to-old-sink, file kept, levels pass through) links the function on every platform, so a bump that drops the patch reds `C++ Tests` at link time; `LlamaLoggerTest` (model-free, needs only `libjllama`: a logger set before a deliberately failing load on a non-GGUF file sees the `srv … loading model` INFO line and llama's ERROR line, in TEXT and JSON) and the re-enabled `LlamaModelTest#testLogText/testLogJSON` plus `#testLoggerSetBeforeLoadSurvivesTheLoad` (vocab-only load) cover the Java side. Upstream-submittable ("common : add a callback sink to common_log for embedding hosts"); **not yet filed upstream**. Touches only `common/log.{h,cpp}`, which no other patch touches. **Refreshed at the b11401 bump** (#29895 adds a `colors` member next to the sink's members and a `get_colors()` next to `set_callback()`; only context moved, the `+`/`-` lines are unchanged, and upstream still has no hook -- it added `common_log_get_colors` only). **On a bump, check whether upstream added a hook of its own (grep `callback` in `common/log.h`) and, if so, DROP this patch and port `setLogger` to theirs rather than refreshing it.** | +| `0012-model-guard-zero-split-sum-and-name-the-device-index.patch` | **A GPU that reports zero free memory makes every model load fail with the unactionable `error loading model: vector`.** `llama_model_base::load_tensors` (`src/llama-model.cpp`) weights the per-device layer split by `ggml_backend_dev_memory()`'s `free`, then normalises: `splits[i] /= split_sum`. With a single device reporting `free == 0` that is `0/0` → **NaN** in every split point; NaN compares false against everything, so the `std::upper_bound` below returns the end iterator, `layer_gpu == n_devices()`, and `devices.at(layer_gpu)` throws `std::out_of_range` — whose libc++ `what()` is the bare string `"vector"`, which `llama.cpp`'s `catch (const std::exception &)` prints verbatim. Upstream's `free == 0 && total == 0` host-memory fallback does **not** fire, because `total` is `recommendedMaxWorkingSetSize` and is non-zero. **Reachable since b10618..b10797**: upstream `8c0b9cd04` ("metal : fix memory query under low-memory conditions", [#27701](https://github.com/ggml-org/llama.cpp/pull/27701)) changed `ggml-metal-device.m` to `*free = *total > cur ? *total - cur : 0`; before that clamp an over-committed device (`currentAllocatedSize > recommendedMaxWorkingSetSize`) *underflowed* to a huge `size_t`, which normalised fine, so the same precondition was harmless. That is why the `Java Tests macOS …` jobs went red at the b10792→b10797 step while every Linux/Windows job stayed green — **and why only a GPU build can fail this way at all**: `act_gpu_layers` is `devices.empty() ? 0 : …`, so with no GPU backend `devices` is empty, every layer returns early on `cpu_dev`, and the `.at()` line is unreachable. **Shape:** the two blocks are lifted out of `load_tensors` into free functions declared in `src/llama-model.h`, purely so they can be driven by a test — the failing state needs a real over-committed GPU and cannot be arranged through any public API. `llama_model_splits_normalize()` carries **the fix**: on `split_sum == 0` it `LLAMA_LOG_WARN`s and falls back to an even split (`splits[i] = float(i+1)/splits.size()`), the only neutral choice when no device can be preferred and exactly right for a single device. `llama_model_splits_select_device()` carries **the diagnostic**: it bounds-checks the index and throws a `std::runtime_error` naming the function, the offloaded layer, the device index, the split-point count **and the split points themselves** — with NaN splits that message prints `nan` and names the cause outright, which is precisely what was missing when this had to be diagnosed by reading source. **A second, backend-independent trigger reaches the same line**, found while writing this up and verified against the unfixed library: `--tensor-split` values are parsed with `std::stof` and never range-checked (`common/arg.cpp`), so `-ts 1,-1` cancels out, `split_sum` is 0 again, the split points become `[inf, -nan]`, and every layer maps one past the last device — on CUDA, Vulkan or ROCm just as much as on Metal, with no memory pressure involved. That is what makes this an ordinary upstream defect rather than a Metal edge case, and the warning names both causes rather than only the memory one. Also adds upstream `tests/test-model-split.cpp` (5 cases in upstream's `testing.h` style) + its `llama_build_and_test` registration. Touches `src/llama-model.{cpp,h}`, `tests/test-model-split.cpp` and `tests/CMakeLists.txt` — **none** of which any other patch touches, so it is independent of all of them. **Refreshed at the b11570 bump** (#30299 links each upstream test against only the libraries it uses: `llama_build_and_test` takes `LINK `, and the registration this patch adds reads `llama_build_and_test(test-model-split.cpp LINK llama)` now -- the carried test includes `../src/llama-model.h` and needs `llama` only, like the other internal-function tests in that block; the `src/llama-model.{cpp,h}` and `tests/test-model-split.cpp` hunks are unchanged, only their offsets moved). Upstream-submittable ("model: fall back to an even split when no device reports free memory"); **not yet filed upstream**. **Runnable guard: `src/test/cpp/test_model_split.cpp`** — a FetchContent subproject builds with `LLAMA_BUILD_TESTS=OFF`, so the upstream test above is applied-but-never-compiled here (same as `0001`'s test). That file drives the same two functions from `jllama_test`, which runs on **every** platform in `C++ Tests`, so a bump that drops this patch fails the build at link time everywhere instead of surfacing as one red macOS Java job. **Verification limit — read before assuming this can be dropped:** the *failing path* still cannot be reached without a GPU backend, so the guard pins the arithmetic (what actually broke), not the end-to-end load; the end-to-end proof is the macOS CI job. On a bump, re-check whether upstream added its own `split_sum == 0` guard (grep `split_sum` in `src/llama-model.cpp`) and **drop this patch rather than refreshing it** if they did — the fail-loud applier detects "does not apply", never "upstream already fixed this". **Upstream fixed the defect on 2026-10-14, one tag after this pin:** [ggml-org/llama.cpp#29677](https://github.com/ggml-org/llama.cpp/pull/29677), commit [`df18bee11`](https://github.com/ggml-org/llama.cpp/commit/df18bee11e96557b07aa7b12db0976f3c7ac9165) ("model: split layers evenly when no device reports free memory"), one line in `load_tensors()`: `splits[i] = split_sum != 0.0f ? splits[i] / split_sum : float(i + 1) / n_devices()` -- the same even-split fallback as `llama_model_splits_normalize()` here, without the warning, without the `select_device` diagnostic and **without a test** (the PR touches `src/llama-model.cpp` only). It is **not in b11612** (`10da8c463`, 15 commits before it on master; `src/llama-model.cpp:1648` there is still the bare `splits[i] /= split_sum`, measured), which is why the patch is still carried at this pin and applies unchanged. **At the first bump past it the drop-check above fires: drop `0012`, do not refresh it.** What that does to the guard is decided in `TODO.md` ("Drop patch `0012` at the first bump past b11612"): `test_model_split.cpp` links the two free functions only this patch adds, so it cannot outlive the patch as it is, and upstream shipped no test to point it at. | +| `0014-common-log-callback-sink.patch` | **Gives `common_log` a callback sink: `common_log_set_callback(log, cb, user_data)` (`common/log.{h,cpp}`).** This is what `LlamaModel.setLogger` hooks. Before it, the Java logger was a `llama_log_set()` callback, which has two holes, both found while chasing `slot print_timing` lines interleaving with the Atmosphere agent's streamed answer: **(a)** every model load runs `common_init()`, which re-points `llama_log_set()` at `common_log_default_callback` (`common.cpp:394`), so `setLogger(…)` *before* `new LlamaModel(…)` silently lost the callback; **(b)** the server's own `SRV_*`/`SLT_*` macros are `LOG_INF` and write straight into `common_log`, which `llama_log_set()` never carried, so the per-request `slot …`/`srv …` lines could not be routed to Java at all (the reason `LlamaModelTest#testLogText/JSON` sat `@Disabled` for years). `common_log` upstream offers file, colors, prefix, timestamps, verbosity and JSONL but no hook. The patch adds one: while a callback is set, the worker thread hands every entry to it **instead of** printing to stdout/stderr (a `--log-file` still receives them); the callback gets the bare formatted message (no prefix/timestamp/colors) with the `ggml_log_callback` signature; swapping the callback pauses the worker first, so queued entries reach the *previous* sink (which is what makes `setLogger(format, null)` a synchronous drain). With the sink, `common_init()`'s `llama_log_set()` reset is harmless — it points at the default callback that feeds `common_log`, i.e. exactly the path into the sink — so the ordering problem (a) disappears without any re-install logic, and the `srv`/`slot` lines (b) arrive because they are `common_log` entries. `jllama.cpp`'s `setLogger` therefore sets `common_log_set_callback(common_log_main(), trampoline)` **plus** `llama_log_set(common_log_default_callback)` (so llama/ggml lines feed `common_log` even before the first load). Two consequences to know: the callback runs on `common_log`'s **worker thread**, a plain `std::thread` llama.cpp re-creates on every pause/resume and never attaches to the JVM — the trampoline attaches it **once per worker thread** as a daemon, named `jllama-log-worker`, and a `thread_local` guard detaches it when the thread exits (`get_jni_env_attaching`; `common_log::pause()` joins the worker, a normal exit on which TLS destructors run, and `JNI_OnUnload` pauses it before anything else). Until 5.2.x it attached and detached per line, which created a `java.lang.Thread` object per log line (13 lines of a failed load, 13 `Thread` objects -- `LlamaLoggerTest` pins the single attach now). Each call runs inside a `PushLocalFrame`/`PopLocalFrame` pair, because a thread that stays attached and has no Java frame would otherwise accumulate local references, `setLogger` must call `common_log_set_callback` **outside** `g_log_mutex`, because the pause joins the worker, which needs that mutex to read the callback, and `setLogger` callers are serialized by a **separate** `g_set_logger_mutex`: two unserialized swaps race on the worker's `std::thread` (one joins it while the other assigns a fresh thread over the still-joinable object = `std::terminate`, the whole JVM), which `LlamaLoggerTest#concurrentSetLoggerCallsDoNotRaceOnTheLogWorker` reproduced before the mutex existed. Two caveats the Javadoc carries: a caller must not hold a lock the *previous* callback needs (the drain runs it on the worker while the caller waits), and the verbosity threshold is process-wide and reset by **every** load (`common_params_parse` ends with `common_log_set_verbosity_thold(params.verbosity)`, default 3), so a load without `-lv` puts it back to 3 — a review assumed the opposite, and `LlamaLoggerTest#verbosityThresholdIsProcessWideAndEveryLoadSetsIt` now pins the measured behaviour. And the verbosity threshold applies *before* the sink: at the default (`3`) the Java logger sees errors, warnings and the server's INFO lines, while llama/ggml INFO lines (`common_log_get_verbosity` maps them to TRACE = 4) arrive only from `setLogVerbosity(4)` on — the same filtering the console gets, and a behaviour change for consumers who captured the unfiltered `llama_log_set()` stream before. **Runnable guards:** `src/test/cpp/test_common_log_callback.cpp` (6 tests over a private `common_log_init()` instance: delivery, bare text under prefix+timestamps, clear, swap-drains-to-old-sink, file kept, levels pass through) links the function on every platform, so a bump that drops the patch reds `C++ Tests` at link time; `LlamaLoggerTest` (model-free, needs only `libjllama`: a logger set before a deliberately failing load on a non-GGUF file sees the `srv … loading model` INFO line and llama's ERROR line, in TEXT and JSON) and the re-enabled `LlamaModelTest#testLogText/testLogJSON` plus `#testLoggerSetBeforeLoadSurvivesTheLoad` (vocab-only load) cover the Java side. Upstream-submittable ("common : add a callback sink to common_log for embedding hosts"); **not yet filed upstream**. Touches only `common/log.{h,cpp}`, which no other patch touches. **Refreshed at the b11401 bump** (#29895 adds a `colors` member next to the sink's members and a `get_colors()` next to `set_callback()`; only context moved, the `+`/`-` lines are unchanged, and upstream still has no hook -- it added `common_log_get_colors` only). **Refreshed at the b11612 bump** (`common_log_set_stdo()`, a stdout/stderr on-off switch stored as a `stdo` member right where the sink's members sit, and `print_entry()` now prints to the console only `if (stdo)`): three hunks of `common/log.cpp` lost their context and were regenerated; the sink keeps precedence -- `if (callback) { callback(...) } else if (stdo) { e.print(); }` -- so a sink still replaces the console and `set_stdo(false)` without a sink still silences it. Still no hook upstream (`common/log.h` names only `common_log_default_callback`). **On a bump, check whether upstream added a hook of its own (grep `callback` in `common/log.h`) and, if so, DROP this patch and port `setLogger` to theirs rather than refreshing it.** | | `0015-rpc-embeddable-client-and-stoppable-server.patch` | **Makes ggml-rpc usable inside a JVM** (see "RPC backend" below). Upstream treats every client-side RPC problem as `GGML_ABORT` — a malformed endpoint, a server that is not running, a failed handshake — which in a JVM kills the application over a typo in `--rpc`. **(1)** The *registration* path reports failure instead: `rpc_dispatcher::start()` returns `false`, `try_get_dispatcher()` `nullptr`, `ggml_backend_rpc_get_device_count()` `0`, `ggml_backend_rpc_add_server()` `nullptr`, and `add_rpc_devices()` (`common/arg.cpp`) throws `std::invalid_argument` naming the server where it used to register `nullptr` and silently run without it. Every path *after* registration keeps the original contract through `get_dispatcher()`, which still aborts — a server that vanishes mid-inference stays fatal (TODO). **(2)** `add_server()`'s cache hit is re-checked (a stopped server used to pass registration and abort on the first tensor upload), and `ggml_backend_rpc_get_device_memory()` of a gone server returns 0/0 instead of aborting — RPC devices stay in ggml's process-wide registry forever, and `common_init()` queries the memory of **every** registered device at `-lv 4`, so a stopped server would otherwise take down an unrelated later load. **(3)** The server becomes stoppable: `ggml_backend_rpc_stop_server()` (disconnects the client being served, wakes `accept()` with a loopback connection — the one portable way) and `ggml_backend_rpc_server_listening()`; the loop now reaches its cleanup and frees its backends on every early return, but deliberately **not** `rpc_transport_shutdown()`, whose `WSACleanup()` would invalidate every other RPC socket in the process. **(4)** transport: `MSG_NOSIGNAL`/`SO_NOSIGPIPE`, `socket_t::shutdown()`, and the fds `connect()`/`create_server()`/`accept()` leaked on their error paths are closed. Touches `ggml/include/ggml-rpc.h`, `ggml/src/ggml-rpc/{ggml-rpc.cpp,transport.cpp,transport.h}` and one function in `common/arg.cpp` (`add_rpc_devices`, away from `0001`'s `common_params_parse` hunks). **Runnable guard: `src/test/cpp/test_rpc.cpp`** — links both new functions (a bump that drops the patch fails `C++ Tests` at link time on every platform) and drives registration, stop and the memory query over loopback. Upstream-submittable; **not yet filed upstream**. On a bump, check whether upstream grew its own stop function or non-aborting registration (grep `stop_server` / `GGML_ABORT("Failed to connect` in `ggml-rpc.cpp`). **Refreshed at the b11450 bump** (#26610, `-sm tensor` over RPC, ~800 lines in `ggml-rpc.cpp`, protocol 7 → 8): three hunks lost their context -- new dispatcher members beside `start()`, and new `get_proc_address` names before `return NULL` (the two new names now sit after `ggml_backend_comm_allreduce_tensor`); the `+`/`-` lines are unchanged. The server-to-server comm #26610 adds (a listener on `0.0.0.0`, an `accept()` a stop does not wake) is on file in `TODO.md`. **Refreshed at the b11558 bump** (#29544 turns `GGML_RPC_DEBUG` into a verbosity level 0-3: one `ggml-rpc/log.h` with `LOG_ERROR`/`LOG_WARN`/`LOG_INFO` and `LOG_DBG`/`LOG_DBG2`/`LOG_DBG3`, every `GGML_LOG_*`, `printf` and `fprintf(stderr)` of the backend routed through it, and `transport_name()` where `shutdown()` goes in `transport.h`): 12 of the 24 hunks lost their context and were regenerated against the new text; the changes are the same, and the patch's own five error lines (the HELLO handshake in `negotiate_hello`, four in `rpc_dispatcher::start()`) use `LOG_ERROR` now like the code around them (`LOG_ERROR` is `GGML_LOG_ERROR`). One consequence outside the patch: the server's `Accepted client connection` line is `LOG_INFO` through ggml's log callback now (stderr by default, a `setLogger` sink at verbosity 4), no longer `printf` to stdout; `smoke-rpc.sh` captures both streams into `rpc-server.log`, so its grep is unaffected. | | `0017-ggml-cpu-x86-prefetch-byte-offset-and-msvc-type.patch` | **Makes `GGML_CPU_ALL_VARIANTS=ON` buildable with plain clang on Windows, and fixes a prefetch address that is wrong everywhere.** The four `_mm_prefetch` calls in `ggml/src/ggml-cpu/arch/x86/quants.c` (the SSSE3-without-AVX branch of `ggml_vec_dot_q4_0_q8_0`, and the only `_mm_prefetch` calls in the whole ggml tree) carry two independent defects; both are fixed by routing the address through one new `static inline` helper in the new header `ggml/src/ggml-cpu/arch/x86/prefetch.h`, which is included relatively, so **no CMake change is needed**. **(1) Type -- fatal only on a current clang:** MSVC declares the intrinsic as `void _mm_prefetch(char const *, int)`; clang's own `xmmintrin.h` defines a casting macro but only `#ifndef _MSC_VER`, so on a `*-windows-msvc` target clang leaves the declaration to Microsoft and a typed pointer is `-Wincompatible-pointer-types`. **clang 22 promoted that to an error by default** ([llvm-project #157364](https://github.com/llvm/llvm-project/pull/157364); the [22.1.0 release notes](https://releases.llvm.org/22.1.0/tools/clang/docs/ReleaseNotes.html) add that `-Wno-error=incompatible-pointer-types` downgrades it) -- *not* clang 16, which promoted only `-Wincompatible-FUNCTION-pointer-types`; an earlier version of this row said otherwise. Measured on the real file at `-msse4.2`, unpatched → patched: clang **20.1.8** 4 warnings → 0/0, clang **22.1.3** 4 errors → 0/0, clang **23.1.3** 4 errors → 0/0. Plain clang is exactly what `cmake/x64-windows-llvm.cmake` selects -- and plain clang is the only way to get all **14** CPU variants, because `clang-cl` makes CMake's `MSVC` true and the `if (NOT MSVC)` block in `ggml/src/CMakeLists.txt` then drops `zen4`, `cooperlake`, `sapphirerapids`, `ivybridge` and `piledriver`. **(2) Arithmetic:** `&x[ib] + sizeof(block_q4_0)` adds `sizeof()` *elements*, not bytes -- +324 and +1156 instead of +18 and +34 (the sizes `ggml-common.h` pins by `static_assert`), and +648 instead of +36 for the `2 *` pair. That half is wrong on every platform, including where the call compiles. **Measured:** no throughput difference (see the test-table row), so this is a correctness and portability fix, not a performance one. **Upstream-submittable and meant to be submitted.** **Why upstream's own build is green is measured, not an open question:** their `windows-cpu` release job builds exactly this with plain clang, `GGML_CPU_ALL_VARIANTS=ON` and no suppressing flag (`CMAKE_ARGS` carries only build-target toggles), and ships `ggml-cpu-sse42.dll` -- because the `windows-2025-vs2026` image has **LLVM 20.1.8** on `PATH`, where these are warnings. That is narrower than it looks: the same image also ships **clang 22.1.3** inside Visual Studio 2026 (`VC/Tools/Llvm/x64`), which *fails* on the unpatched file -- upstream reaches the working compiler only because `C:\Program Files/LLVM/bin` comes first on `PATH`. It is **not** a header difference: all three MSVC toolsets on that image (14.29.30133, 14.44.35207, 14.51.36231) declare it identically as `extern void _mm_prefetch(char const*_A, int _Sel);` -- that was the first hypothesis, and a one-off probe workflow (deleted again once it had answered) disproved it by printing the declaration from each toolset on the image. So this patch is **ahead of** upstream rather than diverging: their build breaks the moment the image ships a clang that makes the diagnostic fatal, and until then they compile four warnings and a prefetch address 18x/34x too far. **Runnable guard: `src/test/cpp/test_prefetch.cpp`.** On a bump, run `git grep -n _mm_prefetch ggml/src/ggml-cpu/arch/x86/quants.c` at the new tag: if the call sites already cast **before** the addition, **drop this patch rather than refreshing it**. | | `0016-model-kolibri1.patch` | **Adds Aleph Alpha's Kolibri-1 (architecture `kolibri1`) ahead of upstream** -- a 78B German/English reasoning MoE (3.46B active) that upstream does not support yet ([ggml-org/llama.cpp#29922](https://github.com/ggml-org/llama.cpp/issues/29922), no PR at the time). A **temporary carry, not a fix**: drop it the moment upstream registers the architecture (`git grep -n kolibri src/llama-arch.cpp` at the new tag), and keep `test_kolibri1.cpp` -- its numerical comparisons must then pass against upstream's implementation, while its GGUF-format rows follow whatever format upstream's converter fixes (see the test-table row). **Architecture** (Aleph Alpha's vLLM reference, `aleph_alpha_inference/kolibri1.py` at `049a6a7`): GQA with per-head q/k RMSNorm; `layer_types` interleaves sliding-window layers (NEOX RoPE) with full-attention layers that have **no** positional encoding; sandwich norms around attention and MoE; every layer routed MoE + one ungated shared expert; the router selects top-k on **`logits + expert_bias`** and weights by the **unbiased `sigmoid(logits)`** -- not DeepSeek-V3's sigmoid router (selection on `sigmoid(logits) + bias`), which picks other experts as soon as the bias is non-zero. The graph therefore builds the selection itself and passes it to `build_moe_ffn` (`selected_experts_in`), leaving upstream's shared MoE code untouched. **Derived from two community ports, which are incompatible with each other's GGUFs** -- Eliasfpv28's AFMoE-based one (base b11378, `expert_gating_func` 2, pre-tokenizer `qwen2`) and the Qwen3-MoE-based one behind Hob-forge's GGUFs (base b11381, as carried in [mjwsolo/localcode#101](https://github.com/mjwsolo/localcode/pull/101): gating 5, pre-tokenizer `kolibri1`, rejects gating 2). This patch **loads both dialects**: gating 2, 5 or absent (all mean the one router), `kolibri1` mapped to the Qwen2 pre-tokenizer in `llama-vocab.cpp`, `expert_weights_norm` / `expert_shared_count` / `output.weight` optional. Every source, with license, is listed in the patch header; `REUSE.toml` annotates the file `MIT AND Apache-2.0` (Eliasfpv28's additions are Apache-2.0). Touches `src/llama-arch.{h,cpp}`, `src/llama-model.cpp` (mapping + NEOX rope list), `src/llama-vocab.cpp` (one name), `src/models/models.h` and the new `src/models/kolibri1.cpp`; no converter. **Runnable guard: `src/test/cpp/test_kolibri1.cpp`** (writes tiny random GGUFs in both dialects and compares every logit with an independent double-precision reference). **Verified against the real 78B model on 2026-10-11** (rented 16-vCPU / 123 GB instance, both published dialects fetched on the machine): both load, and a chat turn through the model's own template answers correctly in each -- `content` `"Berlin."` for a German question and `"51"` for 17x3, with the chain of thought in `reasoning_content` and `finish_reason` `stop`. **The two dialects tokenize identically** (`[452 22090 493 1678 2459 5513 46]`, round-tripping to the input sentence), which is the direct evidence that mapping `kolibri1` to the Qwen2 pre-tokenizer is right. Q4_K_M (44 GB) gave 13.4 t/s and Q3_K_S (31 GB) 14.6 t/s of generation on 16 cores. **Two traps for whoever repeats this:** a *raw* completion on this instruct model degenerates into repetition whatever the patch does, so measure through the chat endpoint; and `chatCompleteText` returns only `content`, which is **empty** when a reasoning model spends its token budget inside the thinking block -- use `handleChatCompletions` and a budget in the hundreds. **Still not verified:** GPU backends -- the community ports report real-model checks (top-1 67/67 against a float64 reference for Q8_0, tool calls, German text). | -| `0018-ggml-amx-free-the-aligned-buffer.patch` | **The AMX weight buffer is freed with `free()` but allocated with `_aligned_malloc()` on Windows -- `STATUS_HEAP_CORRUPTION` (0xC0000374) the moment a model is unloaded.** `ggml_backend_amx_buffer_type_alloc_buffer()` (`ggml/src/ggml-cpu/amx/amx.cpp`) allocates with `ggml_aligned_malloc()`, which is `_aligned_malloc()` under `_MSC_VER` (plain clang on a `*-windows-msvc` target defines it too); `ggml_backend_amx_buffer_free_buffer()` called `free()` on it. Fine on Linux (`posix_memalign` + `free`), fatal on Windows, and only there. The patch frees with `ggml_aligned_free(buffer->context, buffer->size)`, exactly what `ggml_backend_cpu_buffer_free_buffer()` does; on Linux that is `free()`, so nothing changes there. **Reached by** the `sapphirerapids` CPU module (the only variant built with `AMX_TILE AMX_INT8`, which `ggml_backend_load_best` picks on Sapphire/Emerald/Granite Rapids; `ggml_amx_init()` is `return true` on `_WIN32`) and a model whose weights have AMX kernels (Q4_0, Q4_1, Q8_0, Q4_K, Q5_K, Q6_K, IQ4_XS -- i.e. every CI model). **Found by the second full run of the modular natives** (38069595101): `Java Tests Windows 2025 x86_64 (clang)` landed on an `Intel(R) Xeon(R) 6973P-C` runner (Granite Rapids), loaded `ggml-cpu-sapphirerapids.dll`, ran `GrammarExample`'s two generations fine and died with exit code -1073740940 at `LlamaModel.close()`; the first run (38065017595) had the same job green on an AMD EPYC 7763 (no AVX-512, `haswell`). **The Windows runners are a CPU lottery**, so which CPU module a Windows job exercises differs per run; before the modular natives the Windows jar was a static Haswell-level build and this code never ran. **Of the six callers of `ggml_aligned_malloc` in the tree this was the only mismatched pair** (`ggml-backend.cpp`, `ggml-cpu.c` twice, `ggml-openvino.cpp`, `ggml-zdnn.cpp` and `ggml.c` all pair correctly), and `ggml-backend.cpp:2464` is the line the fix copies. **Reproduced deliberately rather than waiting for the lottery again**: on a rented Granite Rapids machine, one CPU module staged at a time with the same model and the DLLs CI itself built, `sapphirerapids` crashed **6 of 6** runs -- including at **one** thread, which rules out a data race -- while `cooperlake`, `icelake` and `haswell` were clean 6 of 6, and on Linux with the same CPU all six exit 0 with byte-identical output. The crash lands *after* the inference prints its timings, in cleanup: the signature of a heap corrupted earlier and detected at `free`. With the patch applied the same matrix on the same machine is **clean in all twelve runs**. **The guard is a configure-time check, not a `jllama_test` case**: the AMX code exists only in that one variant of a `GGML_CPU_ALL_VARIANTS` build and executes only on AMX hardware, which the static test build, the local agent's Zen 3 and the cloud sandboxes lack -- so **`llama/cmake/check-aligned-free-pairing.cmake`** fails the configure on **every** platform when a ggml source calls `ggml_aligned_malloc` and never `ggml_aligned_free` (verified red against the unpatched source, naming `amx.cpp`, and green with this patch). A run-time test cannot replace it: without an AMX CPU there is nothing to observe, which is exactly how this shipped. What exercises it: the Windows x86-64 Java test job and the `windows-x86-64` smoke row whenever GitHub hands them an Intel AMX runner; the patch itself is held by `check-patches.py`, applies at b11574, and the patched file compiles with the variant's flags (GCC 13, `-mamx-tile -mamx-int8 -mavx512vnni ...`). Upstream master still carried the `free()` on 2026-10-10; **not yet filed upstream** (nothing before the release). **Drop-check on every bump:** `grep -n "free(buffer->context)" ggml/src/ggml-cpu/amx/amx.cpp` at the new tag -- gone means upstream fixed it, drop the patch. | +| `0018-ggml-amx-free-the-aligned-buffer.patch` | **The AMX weight buffer is freed with `free()` but allocated with `_aligned_malloc()` on Windows -- `STATUS_HEAP_CORRUPTION` (0xC0000374) the moment a model is unloaded.** `ggml_backend_amx_buffer_type_alloc_buffer()` (`ggml/src/ggml-cpu/amx/amx.cpp`) allocates with `ggml_aligned_malloc()`, which is `_aligned_malloc()` under `_MSC_VER` (plain clang on a `*-windows-msvc` target defines it too); `ggml_backend_amx_buffer_free_buffer()` called `free()` on it. Fine on Linux (`posix_memalign` + `free`), fatal on Windows, and only there. The patch frees with `ggml_aligned_free(buffer->context, buffer->size)`, exactly what `ggml_backend_cpu_buffer_free_buffer()` does; on Linux that is `free()`, so nothing changes there. **Reached by** the `sapphirerapids` CPU module (the only variant built with `AMX_TILE AMX_INT8`, which `ggml_backend_load_best` picks on Sapphire/Emerald/Granite Rapids; `ggml_amx_init()` is `return true` on `_WIN32`) and a model whose weights have AMX kernels (Q4_0, Q4_1, Q8_0, Q4_K, Q5_K, Q6_K, IQ4_XS -- i.e. every CI model). **Found by the second full run of the modular natives** (38069595101): `Java Tests Windows 2025 x86_64 (clang)` landed on an `Intel(R) Xeon(R) 6973P-C` runner (Granite Rapids), loaded `ggml-cpu-sapphirerapids.dll`, ran `GrammarExample`'s two generations fine and died with exit code -1073740940 at `LlamaModel.close()`; the first run (38065017595) had the same job green on an AMD EPYC 7763 (no AVX-512, `haswell`). **The Windows runners are a CPU lottery**, so which CPU module a Windows job exercises differs per run; before the modular natives the Windows jar was a static Haswell-level build and this code never ran. **Of the six callers of `ggml_aligned_malloc` in the tree this was the only mismatched pair** (`ggml-backend.cpp`, `ggml-cpu.c` twice, `ggml-openvino.cpp`, `ggml-zdnn.cpp` and `ggml.c` all pair correctly), and `ggml-backend.cpp:2464` is the line the fix copies. **Reproduced deliberately rather than waiting for the lottery again**: on a rented Granite Rapids machine, one CPU module staged at a time with the same model and the DLLs CI itself built, `sapphirerapids` crashed **6 of 6** runs -- including at **one** thread, which rules out a data race -- while `cooperlake`, `icelake` and `haswell` were clean 6 of 6, and on Linux with the same CPU all six exit 0 with byte-identical output. The crash lands *after* the inference prints its timings, in cleanup: the signature of a heap corrupted earlier and detected at `free`. With the patch applied the same matrix on the same machine is **clean in all twelve runs**. **The guard is a configure-time check, not a `jllama_test` case**: the AMX code exists only in that one variant of a `GGML_CPU_ALL_VARIANTS` build and executes only on AMX hardware, which the static test build, the local agent's Zen 3 and the cloud sandboxes lack -- so **`llama/cmake/check-aligned-free-pairing.cmake`** fails the configure on **every** platform when a ggml source calls `ggml_aligned_malloc` and never `ggml_aligned_free` (verified red against the unpatched source, naming `amx.cpp`, and green with this patch). A run-time test cannot replace it: without an AMX CPU there is nothing to observe, which is exactly how this shipped. What exercises it: the Windows x86-64 Java test job and the `windows-x86-64` smoke row whenever GitHub hands them an Intel AMX runner; the patch itself is held by `check-patches.py`, applies at b11612, and the patched file compiles with the variant's flags (GCC 13, `-mamx-tile -mamx-int8 -mavx512vnni ...`). Upstream master still carried the `free()` on 2026-10-10; **not yet filed upstream** (nothing before the release). **Drop-check on every bump:** `grep -n "free(buffer->context)" ggml/src/ggml-cpu/amx/amx.cpp` at the new tag -- gone means upstream fixed it, drop the patch. | | `0019-ggml-cpu-select-the-neon-fp32-to-fp16-conversion.patch` | **Every ARM build converts fp32 to fp16 in software, ours and upstream's.** The `__ARM_NEON` branch of `simd-mappings.h` defines both `GGML_CPU_COMPUTE_*` macros and selects only `FP16_TO_FP32`, so `GGML_CPU_FP32_TO_FP16` falls through to `GGML_COMPUTE_FP32_TO_FP16` -- ggml-base's deliberately portable software bit manipulation -- while `neon_compute_fp32_to_fp16`, one `fcvt`, stays defined and referenced by nothing. The `GGML_CPU_*` pair exists to be the *accelerated* variant; on ARM it was defined to be the unaccelerated one. `__POWER9_VECTOR__` and `__riscv_zfhmin` select both, and the comment on that fallback has always promised NEON does too. **Not a regression** -- absent in all 25 most recent commits touching the file, back to 2025-04-10. **Why only ARM feels it:** `ggml_cpu_fp32_to_fp16`, the `from_float` of `GGML_TYPE_F16`, has vector paths for F16C, RISC-V and s390x (the last from #30157) and **none** for NEON, so there the whole function is the scalar loop through the macro. **In the shipped artifacts:** no `fcvt` of any form in that function in any of this project's 17 ARM libraries (`cpu-linux-aarch64` 8 variants, `cpu-android-aarch64` 7, `cpu-windows-aarch64`, `metal-macos-aarch64`) nor in upstream's own `bin-ubuntu-arm64` / `bin-macos-arm64`; a plain `gcc` with no flags defines `__ARM_NEON` and `__ARM_FP16_FORMAT_IEEE` on aarch64, so nothing can miss it. **Measured 2.45x-13.3x** on five ARM core types, three operating systems and nine compiler builds, with the opposite direction in the same run as a control (within 3.2 % everywhere); **bit-exact over all 2^32 float inputs**, differing only in NaN payload, where x86 F16C gives the same numbers -- so it makes ARM agree with what x86 already ships. **Not an inference speed-up:** `perf` puts 83.9 % of an F16 `pp512` run in tinyBLAS's gemm, which converts in its own microkernel, and 0.03 % here. The patch also moves #30270's two F16C selections from 88 lines below the chain into the `__F16C__` branch (behaviour-preserving, proven at object level) and puts an `#error` pair in their place that refuses a branch defining a COMPUTE macro without selecting it -- its position chosen by a mutation measurement, not by taste (4 of 4 correctness cases caught before the fallback against 3 of 4 directly after the chain). Full record: [`docs/benchmarks/arm-fp16-conversion.md`](docs/benchmarks/arm-fp16-conversion.md). **Submitted upstream as [ggml-org/llama.cpp#30505](https://github.com/ggml-org/llama.cpp/pull/30505)** (open), byte-identical to this diff. **Drop-check on every bump:** `git grep -n "GGML_CPU_FP32_TO_FP16(x) GGML_CPU_COMPUTE_FP32_TO_FP16" ggml/src/ggml-cpu/simd-mappings.h` at the new tag -- if the `__ARM_NEON` branch carries that line, **drop this patch rather than refreshing it**. | **Dropped patches** (`0004`, `0005`, `0009`, `0010`, `0011`, `0013` -- upstream fixed each defect @@ -1721,7 +1721,7 @@ git show :tools/server/server-task.cpp | { grep -oE '\{ *"[A-Za-z_0-9.]+" * | File | What to watch for | |------|-------------------| | `common/common.h` | `common_params`/`common_params_speculative` struct fields, `model_alias` container type, `common_init_result` shape, `build_info` symbol (removed in b8831 — now `llama_build_info()` from `build-info.h`) | -| `common/chat.h` | `common_chat_session` (b11531, #30210: one object per generation owning the prompt and the parser state; `oaicompat_chat_params_parse` fills it as its last argument, `server_task::apply_chat_session` consumes it, `server_response_reader::post_tasks` takes it -- `jllama.cpp` threads one through every chat entry point), `common_chat_parser_params` (was `common_chat_syntax`), `to_json_oaicompat`, `common_chat_msg_diff_to_json_oaicompat`, `set_tool_call_ids` | +| `common/chat.h` | `common_chat_prompt` (b11612, #30389 "chat: replace output string with parts": `common_chat_params::prompt` and `common_chat_session::prompt()` are a list of template / user-input parts now, `str()` renders the text, `common_chat_template_direct_apply` returns one -- `jllama.cpp` never reads the prompt, so nothing here changed), `common_chat_session` (b11531, #30210: one object per generation owning the prompt and the parser state; `oaicompat_chat_params_parse` fills it as its last argument, `server_task::apply_chat_session` consumes it, `server_response_reader::post_tasks` takes it -- `jllama.cpp` threads one through every chat entry point), `common_chat_parser_params` (was `common_chat_syntax`), `to_json_oaicompat`, `common_chat_msg_diff_to_json_oaicompat`, `set_tool_call_ids` | | `common/speculative.h` | `common_speculative_init`, `common_speculative_draft`, `common_speculative_accept` signatures, struct names | | `tools/mtmd/mtmd.h` | `mtmd_context_params` fields, `image_marker`/`media_marker` API, deprecated symbols (was `common/mtmd.h` before ~b8190) | | `include/llama-cpp.h` | `common_init_result_ptr` type, access pattern changes (`.get()` vs `->method()`) | @@ -2395,7 +2395,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" | `src/test/cpp/test_server.cpp` | 206 | Upstream result types: `server_slot_stats` (the `timings` JSON payload; replaced `result_timings` in b10408), `task_params::to_json()` (incl. `dry_sequence_breakers`, `preserved_tokens`, `timings_per_token`), `completion_token_output`, `server_task_result_cmpl_partial` (non-oaicompat + `to_json_oaicompat` + logprobs + `to_json_oaicompat_chat` + `to_json_anthropic` + dispatcher), `server_task_result_cmpl_final` (non-oaicompat + `to_json_oaicompat` + `to_json_oaicompat_chat` + `to_json_oaicompat_chat_stream` + `to_json_anthropic` + `to_json_anthropic_stream` + tool_calls + dispatcher), `server_task_result_embd`, `server_task_result_rerank`, `server_task_result_metrics` (`to_metrics()` = the `/metrics` Prometheus exposition text; its `to_json()` has been unused since b10519 and returns `json{}` = JSON null), `server_task_result_slots` (`to_json()` = the `/slots` array, fed by the b10519 `SERVER_TASK_TYPE_SLOT_GET` task), `server_task_result_slot_save_load`, `server_task_result_slot_erase`, `server_task_result_apply_lora`, `server_task_result_get_lora`, `server_task_result_error`, `format_error_response`, `server_task::need_sampling()`, `server_task::n_tokens()`, `server_schema::eval_llama_cmpl_schema()` (parsing pipeline + grammar routing + error paths + per-request `dry_*` and `sse_ping_interval` field round-trips incl. hard-limit + server-default inheritance), `response_fields` projection | | `src/test/cpp/test_json_helpers.cpp` | 71 | All functions in `json_helpers.hpp`: `get_result_error_message`, `get_result_error_code` (the HTTP code of a failed task result, from which `jllama.cpp` picks `InvalidRequestException` or `LlamaException`), `results_to_json`, `rerank_results_to_json` (incl. missing/out-of-range `index` rejection), `parse_encoding_format`, `extract_embedding_prompt`, `is_infill_request`, `parse_slot_prompt_similarity`, `parse_positive_int_config`, `wrap_stream_chunk`, `server_metrics_to_json`, `route_error_message` | | `src/test/cpp/test_log_helpers.cpp` | 13 | All functions in `log_helpers.hpp`: `log_level_name`, `format_log_as_json` | -| `src/test/cpp/test_common_log_callback.cpp` | 6 | **The runnable guard for `patches/0014`**: `common_log_set_callback()` on a private `common_log_init()` instance (never `common_log_main()`, so the process-wide logger the other tests print through is untouched) — delivery of level + bare text, no prefix/timestamp even when both are on (what `common_init()` does), clearing stops delivery, a swap drains queued entries to the *previous* sink (the property behind `LlamaModel.setLogger(format, null)` being a synchronous flush), a `--log-file` keeps being written alongside the sink, and every `ggml_log_level` passes through unchanged. The Java half (`LlamaLoggerTest`, model-free) proves the JNI trampoline on top of it. | +| `src/test/cpp/test_common_log_callback.cpp` | 7 | **The runnable guard for `patches/0014`**: `common_log_set_callback()` on a private `common_log_init()` instance (never `common_log_main()`, so the process-wide logger the other tests print through is untouched) — delivery of level + bare text, no prefix/timestamp even when both are on (what `common_init()` does), clearing stops delivery, a swap drains queued entries to the *previous* sink (the property behind `LlamaModel.setLogger(format, null)` being a synchronous flush), a `--log-file` keeps being written alongside the sink, every `ggml_log_level` passes through unchanged, and the sink is not gated on upstream's `common_log_set_stdo()` console switch (b11612: entries reach the callback with stdout/stderr output off). The Java half (`LlamaLoggerTest`, model-free) proves the JNI trampoline on top of it. | | `src/test/cpp/test_jni_helpers.cpp` | 70 | All functions in `jni_helpers.hpp` using a zero-filled `JNINativeInterface_` mock (incl. the `utf8_to_jstring_impl` byte-array string path: emoji byte-preservation, truncated-UTF-8 replace-not-throw). Seven of them pin `jni_guard_impl` — the JNI exception boundary every `Java_*` entry point runs inside — including the `catch (...)` arm that is the only backstop for a non-`std::exception` type, and its two refusals (never `ThrowNew` over a pending Java exception, never with a null class). | | `src/test/cpp/test_tts_wav.cpp` | 2 | The in-memory WAV writer `pcm_to_wav16_bytes` in `tts_wav.hpp` (WAV header/payload + little-endian clamping) — our own code, not upstream. The Qwen3-TTS pipeline it pairs with (`mtmd_helper::gen_audio`) is entirely upstream-owned (no project-side DSP to unit-test here). The load path is additionally covered by `test_tts_params.cpp` (3 tests over `tts_params.hpp`'s `build_tts_params`, plus 2 pinning the upstream `-1` default it depends on), which pins the CPU-thread resolution whose absence used to crash the JVM on every platform — see the `TODO.md` entry for the mechanism. End-to-end coverage is `TtsIntegrationTest`, which is model-gated. | | `src/test/cpp/test_tts_params.cpp` | 13 | The **three** builders every hand-assembled `common_params` goes through: `build_tts_params` (`tts_params.hpp`), `build_train_params` (`train_params.hpp`) and the shared `jllama::resolve_cpu_params` (`cpu_params.hpp`). Each builder is guarded separately on purpose — testing the resolver alone does **not** cover its call sites, because `train_engine.cpp` is compiled into `jllama` only, never into `jllama_test`, and `LlamaTrainerIntegrationTest` is gated on `net.ladenthin.llama.train.model`, which no CI job sets. Without these the JVM-abort bug could regress in the trainer on every platform, unseen. | @@ -2408,11 +2408,11 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" | `src/test/cpp/test_native_server_shutdown.cpp` | 3 | **The runnable guard for the shutdown-handler guard in `patches/0006`/`0007`.** Runs the real `llama_server()` in router mode over an empty `--models-dir` on an ephemeral loopback port (no model, no worker) and stops it the way `native_server.cpp` does. Pins a clean stop (exit code 0), that `llama_server_request_shutdown()` **after** the server returned is a no-op (the deterministic form of the CI `SIGSEGV` — it crashed every run before the fix), and that requests hammered from another thread while the server tears down are safe (5 rounds). Compiled on non-Android only, like `server.cpp` itself. | -**Current total: 607 tests (all passing).** +**Current total: 608 tests (all passing).** #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b11574`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b11612`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.18.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index c4b55b029..ad375739c 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b11574](https://img.shields.io/badge/llama.cpp-%23b11574-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b11574) +[![llama.cpp b11612](https://img.shields.io/badge/llama.cpp-%23b11612-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b11612) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/TODO.md b/TODO.md index 75fbbc3d5..c258d158c 100644 --- a/TODO.md +++ b/TODO.md @@ -842,6 +842,57 @@ whose "17 issues found" status once posted on PRs #248/#298; it posts nothing on never set up deliberately. Whether the project wants one, with which policy, is the owner's call; nothing here blocks on it. +### Drop patch `0012` at the first bump past b11612 (upstream merged the fix as #29677) + +**The drop-check of `0012` fires at the *next* bump, not at the b11612 one.** On 2026-10-14 upstream +merged [ggml-org/llama.cpp#29677](https://github.com/ggml-org/llama.cpp/pull/29677), commit +[`df18bee11`](https://github.com/ggml-org/llama.cpp/commit/df18bee11e96557b07aa7b12db0976f3c7ac9165) +("model: split layers evenly when no device reports free memory"), one line in `load_tensors()`: + +```cpp +splits[i] = split_sum != 0.0f ? splits[i] / split_sum : float(i + 1) / n_devices(); +``` + +That is the even-split fallback `patches/0012` carries in `llama_model_splits_normalize()`, 15 commits +**after** b11612 (`10da8c463`) on master -- at b11612 `src/llama-model.cpp:1648` is still the bare +`splits[i] /= split_sum` (measured on the pristine tag), so the patch is still carried at this pin and +applies unchanged. The PR touches `src/llama-model.cpp` only: no warning, no diagnostic on the device +lookup, **no test**. + +**The rule for the bump that crosses it:** + +1. `git grep -n split_sum src/llama-model.cpp` at the new tag. The guarded line is there → **drop + `0012`, do not refresh it.** The applier will not say so: `0012` applies *next to* upstream's line + (its hunks lift the two blocks out of `load_tensors()` and touch other lines), so it keeps applying + cleanly -- the `0010` case, "still applies, no longer needed". Delete the patch file; its `CLAUDE.md` + row moves to `docs/history/dropped-llama-patches.md` with the note below. +2. What goes with it, deliberately: the `LLAMA_LOG_WARN` naming the cause, the bounds check in + `llama_model_splits_select_device()` whose message names layer, index and split points (upstream keeps + `devices.at()`, so a malformed split still reads `"vector"`), and the never-compiled upstream + `tests/test-model-split.cpp`. None of that is worth a patch on its own. One thing upstream's line does + **not** cover, for the record: a NaN weight (`-ts nan,1` -- `std::stof` accepts it, `NaN != 0.0f` is + true) still produces NaN split points and the bare `"vector"`; `0012`'s message named it. If that ever + matters it is a separate upstream issue, not a reason to keep the patch. +3. **The guard `src/test/cpp/test_model_split.cpp` cannot stay as it is.** It links + `llama_model_splits_normalize` and `llama_model_splits_select_device`, which exist only through the + patch -- without it `jllama_test` fails to link on every platform. That was the design (a dropped + patch reds `C++ Tests`), and at the drop it works against us; and upstream shipped no test the guard + could be re-pointed at. So the guard is replaced by one over upstream's contract, like `0010` / `0011`: + - **Preferred: an end-to-end test through the real load path**, the only public route to the fixed + line. The backend-independent trigger the `0012` row documents is `--tensor-split 1,-1`, which needs + **two** devices: two loopback RPC servers serving the CPU (`test_rpc.cpp`'s `loopback_server`) + register two GPU-type devices; load a tiny GGUF written with the `gguf` API (as `test_kolibri1.cpp` + does) with `tensor_split = {1, -1}` and every layer offloaded. At an unfixed tag the load fails + (`std::out_of_range` from `devices.at()`, the bare `"vector"` in the log); at a fixed one it loads + with the layers split evenly. That pins upstream's line on every platform in `C++ Tests` without a + patch. **Verify first** that two `ggml_backend_rpc_start_server` instances coexist in one process + (`0015`'s stop mechanism is global state); if they cannot, this option is out. + - **Otherwise the guard goes with the patch**, and the drop note says why no unit-level guard is + possible (the line sits inside a member function with no seam). Do **not** keep a refactor-only + residue of `0012` just so the test keeps linking: that is a patch to refresh on every bump for a line + upstream owns, the exact thing the drop rule exists to avoid. +4. `patches/0012`'s header and `test_model_split.cpp`'s header carry the same pointer to this section. + ### Upstream submissions -- not before the release (policy) **Nothing is filed upstream before the release is out**, neither issues nor PRs; the drafts are not @@ -885,15 +936,16 @@ treat an unknown flag as pass-through). So the measurement stands as recorded an to add this case to one of those issues, or wait. Re-check on the next sccache release before spending a CI run on it. -There are **twelve** patches today (`0001`-`0003`, `0006`-`0008`, `0012`, `0014`-`0018`). Each +There are **thirteen** patches today (`0001`-`0003`, `0006`-`0008`, `0012`, `0014`-`0019`). Each accepted upstream change deletes a patch from the bump checklist once the pin is bumped past it. - **Submittable as they are** (self-contained, no jllama specifics): `0001` (Windows arg-parse embed guard, already reported as ggml-org/llama.cpp#26416 with the reproducer in `docs/upstream-investigation-win32-argv-substitution.md`; the maintainers have not picked a direction), `0002` (preserve the caller's load-progress callback), `0006` (embeddable - `llama_server`), `0007` (`llama_server_attach`), `0008` (`LLAMA_SERVER_WORKER_CMD`), `0012` (zero - split-sum guard, ships `tests/test-model-split.cpp`), `0014` (`common_log_set_callback`), `0015` + `llama_server`), `0007` (`llama_server_attach`), `0008` (`LLAMA_SERVER_WORKER_CMD`), ~~`0012`~~ (zero + split-sum guard -- **fixed upstream independently on 2026-10-14 by #29677**, nothing left to submit; + dropped at the first bump past b11612, see the section right above), `0014` (`common_log_set_callback`), `0015` (non-aborting RPC registration, stoppable server, transport fixes), `0017` (x86 prefetch helper), **`0018`** (the AMX buffer released with `free()` instead of `ggml_aligned_free` -- one line, with the 6/6-against-0/6 measurement in its header and a configure-time guard that is red without it; diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 7065361c6..cd69e3c52 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -886,3 +886,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b11570–b11573 | patches + upstream verification | All twelve patches apply (`git apply` in order on a pristine b11573 worktree): `0016` (kolibri1, `src/llama-arch.{h,cpp}`, `src/llama-model.cpp`, `src/models/models.h`) applies next to the new architecture's entries at their own positions, `0012` (refreshed in the previous step) next to the model additions in `src/llama-model.cpp`. Drop-checks against pristine b11573/b11574: `src/llama-model.cpp` changed in the range and still sums and divides by `split_sum` unguarded (`:1642-1648`, `0012`); no `kolibri` under `src/` (`0016`); `common/log.h` still names only `common_log_default_callback` (`0014`); no `stop_server`, the connect failure still aborts at `ggml-rpc.cpp:645` (`0015`); the four `_mm_prefetch` calls at `quants.c:782-801` still uncast (`0017`); `amx.cpp:46` still `free(buffer->context)` (`0018`); `arg.cpp` still carries the `GetCommandLineW` override and no `common_params_parse_main` (`0001`); `server-context.cpp:1212` still assigns the load-progress callback unconditionally (`0002`); no `LLAMA_SERVER_WORKER_CMD` (`0008`), no `get_slot_prompt_similarity` (`0003`) upstream. | | b11573–b11574 | 1 commit, 10 files, ~822/128 lines (58 KiB, the target of the series; no `tools/ui` change). **`ggml/src/ggml-hexagon/`** (#30341 "hexagon: optimize CPY/CONCAT — round 3, short-rows edition"): the Qualcomm Hexagon HTP backend's copy and concat ops gain a gather-rows path for short rows (`htp/gather-rows.{c,h}`, new; `cpy-ops` / `concat-ops` dispatch to it), the backend's own `htp/CMakeLists.txt` lists the file, and `scripts/snapdragon/build.py` grows options -- a backend this project does not build (no `hexagon` row in `natives.csv`; the Android jars carry the CPU modules and OpenCL only), so nothing in the range is compiled here. Root `CMakeLists.txt`, `tools/server/`, `common/`, `src/` and every patch-target file are outside the range. **No project source change.** | | b11573–b11574 | patches + upstream verification | **All twelve patches apply unchanged at the target** (`check-patches.py`: 12 patches, 99 hunks, 0 problems; the real applier `apply-llama-patches.cmake` on a fresh `FetchContent` checkout of b11574 -- `rm -rf build`, so the fetch, the twelve patches and the stamp (`head 8bb3cf0d2`, 58 modified files in the source tree) went through the production path -- then the static test build, `ctest` 607/607, and the model-free native Java tests over it: `NativeLibraryLoadSmokeTest` (the `LlamaCppVersion` pin against the linked build's `b11574-`), `LlamaLoggerTest`, `RpcServerTest`, 19/19). **The modular build** (`-DJLLAMA_BACKEND_DL=ON`, the shape of every Linux / Windows / Android library jar) **is where #30014's hidden visibility (b11570 row) could have bitten, and did not**: `libggml-base.so` exports 689 functions and `libggml.so` 16 now, every one `GGML_API` / `GGML_BACKEND_API`, among them the five the load path needs (`ggml_backend_load_all_from_path`, `ggml_backend_load`, `ggml_backend_reg_get_proc_address`, `ggml_backend_dev_count`, `ggml_log_set`); the CPU modules and `ggml-rpc` are `MODULE` targets outside the preset and export their entry points as before (390 in `libggml-cpu-haswell.so`, 77 in `libggml-rpc.so`); `JNI_OnLoad` registered RPC and CPU from the extraction directory (`2 backend(s) registered ... [RPC, CPU]`, `cascadelake` chosen on the sandbox's Xeon), and `RpcServerTest`, `LlamaLoggerTest`, `BackendLoadTest` and `NativeLibraryLoadSmokeTest` ran 27/27 over the modular library -- the RPC server resolved through `ggml_backend_reg_get_proc_address` included. Drop-checks against pristine b11574 are those of the b11573 row, the range touching no patch-target file: `split_sum` unguarded (`0012`), no `kolibri` (`0016`), no `common_log` hook (`0014`), no `stop_server` (`0015`), uncast `_mm_prefetch` (`0017`), `free(buffer->context)` (`0018`), the `GetCommandLineW` override and no `common_params_parse_main` (`0001`), the unconditional load-progress assignment (`0002`), no `LLAMA_SERVER_WORKER_CMD` (`0008`), no `get_slot_prompt_similarity` (`0003`). `check-natives.py`: 23 natives jars, 0 disagreements. Not run locally: the model-backed Java suite (no GGUF in the sandbox) -- CI's test jobs cover it. | +| b11574–b11612 | 38 commits, 287 files, ~27,300/17,750 lines, 6.4 MB -- bumped in one step rather than in the runbook's < 100 KiB chunks because the bulk is data: `docs/ops/zDNN.csv` (31,433 lines), `src/unicode-data.cpp` (+3,985, #30390 opt-in NFC normalization for Unigram tokenizers), the consolidated context-state tests (#30023: `tests/test-save-load-state.cpp`, `test-recurrent-state-rollback.cpp` and `test-state-restore-fragmented.cpp` become `tests/test-llama-context.cpp` + `tests/llama-context/`), kleidiai (#29219, SME2 Q4_K/Q6_K, `KLEIDIAI_COMMIT_TAG` v1.24.0 → v1.31.0 -- ARM only, not built here) and the Vulkan/CUDA/Hexagon/zDNN backends; `tools/ui` unchanged. **Priority list:** `common/chat.h` (#30389 "chat: replace output string with parts"): `common_chat_params::prompt` and `common_chat_session::prompt()` are a `common_chat_prompt` -- a vector of `{text, is_input}` parts, `str()` renders it, `append()` / `replace_last()` skip user-input parts -- and `common_chat_template_direct_apply()` returns one; `server-common.cpp` reads `out_session.prompt().str()`. `jllama.cpp` threads the session through but never reads the prompt, so **no project source change**. `common/log.{h,cpp}` (#30023, of all things): `common_log_set_stdo()` + a `stdo` member, console output gated on it -- context of `0014`, see the patches row. `include/llama.h`: a comment on `embeddings` only; behind it #30445 ("graph: skip the LM head for pooled embeddings"): a pooled context (`pooling_type != NONE`) no longer computes logits when `embeddings` is on, `llama_set_embeddings()` toggles the graph shape (`sched_need_reserve`), and `has_logits = !cparams.embeddings || graph_has_logits` -- the embedding path only, which `LlamaEmbeddingsTest` and the langchain4j integration job exercise in CI. `tools/server/`: #30243 (`server_decision_escape_special_tokens`, `<|name|>` written with U+00A6 instead of `std::regex`) -- the decision path, no schema or response-key change (`server-schema.cpp` field set and `server-task.cpp` emit keys identical at both tags, the two mechanical checks). `src/llama-model.cpp` (#30405 "do not map foreign ranges into a backend buffer": `get_mapping_ranges()` replaces `get_mapping_range()`, `ctx_buf_maps` becomes `ctx_bufs`; #30445's `build_logits()`; `llama_model_set_attn_rot`): outside `0012`'s and `0016`'s hunks, both apply unchanged. `ggml/src/ggml-backend-reg.cpp` (#28367): `ggml_backend_load_best()` drops the pre-checks on each search path and reports an enumeration error once after the loop -- `JNI_OnLoad` calls `ggml_backend_load_all_from_path()`, unaffected. `ggml/src/ggml-cpu/arch/x86/quants.c` (+64, #30447 AVX2 Q2_0 dot product): away from `0017`'s four `_mm_prefetch` lines. Toolchain pins in upstream's `release.yml` unchanged: OpenVINO `2026.4.1` / `2026.4.1.22982.07f9c262b05`, `GGML_CUDA_CCCL_VERSION=v3.4.3`, ROCm `10.0.0`; BoringSSL `0.20260929.0`; `vendor/` untouched (cpp-httplib 0.61.0, #30494, is after b11612). Root `CMakeLists.txt` unchanged, `GGML_LLAMAFILE_DEFAULT` / `GGML_CUDA_GRAPHS_DEFAULT` still the only ggml defaults it sets. **No project source change.** | +| b11574–b11612 | patches + upstream verification | **Three of the thirteen patches needed a refresh, none was dropped.** `0001`: `tests/test-state-restore-fragmented.cpp` is gone (#30023 consolidated the context-state tests into `tests/test-llama-context.cpp`), so its hunk was dropped (36 → 35 `diff --git` headers); the consolidated test has the `--models` / `filtered_argv` `main()` and by the patch's own rule keeps `common_params_parse`. `0014`: #30023 also added `common_log_set_stdo()` and a `stdo` member exactly where the sink's members and `print_entry()`'s console call sit -- three hunks regenerated, the sink keeps precedence (`if (callback) ... else if (stdo) e.print()`), pinned by a seventh case in `test_common_log_callback.cpp`. `0006` failed only in the cascade behind `0001` (its context is the post-`0001` tree) and applies unchanged once `0001` does. The other ten apply unchanged. `check-patches.py`: 13 patches, 101 hunks, 0 problems; the real applier on a fresh `FetchContent` checkout of b11612 (stamp head `10da8c463`, 57 modified files plus the stamp), then the static test build, `ctest` **608/608**, and the model-free native Java tests over the library (`NativeLibraryLoadSmokeTest` with the `LlamaCppVersion` pin against the linked `b11612-`, `LlamaLoggerTest`, `RpcServerTest`, `BackendLoadTest`: 27/27). **Drop-checks against pristine b11612, and the one that will fire next:** `src/llama-model.cpp:1648` is still the bare `splits[i] /= split_sum` (`0012`) -- **but upstream merged the fix the same day, 15 commits past b11612**: [#29677](https://github.com/ggml-org/llama.cpp/pull/29677) / `df18bee11` ("model: split layers evenly when no device reports free memory"), the identical even-split fallback and no test. The first bump past it drops `0012`; the plan for its guard `test_model_split.cpp`, which links the two functions only the patch provides, is in `TODO.md` ("Drop patch `0012` at the first bump past b11612"). The rest unchanged: no `kolibri` under `src/` (`0016`); `common/log.h` has a `set_stdo` but no hook, only `common_log_default_callback` (`0014`); no `stop_server`, the connect failure still aborts at `ggml-rpc.cpp:645` (`0015`); the four `_mm_prefetch` calls at `quants.c:846-865` still uncast (`0017`); `amx.cpp:46` still `free(buffer->context)` (`0018`); the `__ARM_NEON` branch of `simd-mappings.h` still selects the software conversion (`0019`); `arg.cpp:1288` still carries the `GetCommandLineW` override and no `common_params_parse_main` (`0001`); `server-context.cpp:1212` still assigns the load-progress callback unconditionally (`0002`); no `LLAMA_SERVER_WORKER_CMD` (`0008`), no `get_slot_prompt_similarity` (`0003`). `check-natives.py`: 23 natives jars, 0 disagreements; spotless clean. Not run locally: the model-backed Java suite (no GGUF in the sandbox) and the modular build -- CI's test and smoke jobs cover both. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index ed03d7797..454a5dab2 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -337,7 +337,7 @@ endif() FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b11574 + GIT_TAG b11612 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/patches/0001-win32-arg-parse-embed-guard.patch b/llama/patches/0001-win32-arg-parse-embed-guard.patch index 5a11ce490..ae919871a 100644 --- a/llama/patches/0001-win32-arg-parse-embed-guard.patch +++ b/llama/patches/0001-win32-arg-parse-embed-guard.patch @@ -300,19 +300,6 @@ index 46ded1398..d5f1cabe7 100644 return 1; } -diff --git a/tests/test-state-restore-fragmented.cpp b/tests/test-state-restore-fragmented.cpp -index d5548afba..95ba2b67b 100644 ---- a/tests/test-state-restore-fragmented.cpp -+++ b/tests/test-state-restore-fragmented.cpp -@@ -24,7 +24,7 @@ int main(int argc, char ** argv) { - - common_init(); - -- if (!common_params_parse(argc, argv, params, LLAMA_EXAMPLE_COMMON)) { -+ if (!common_params_parse_main(argc, argv, params, LLAMA_EXAMPLE_COMMON)) { - return 1; - } - diff --git a/tests/test-thread-safety.cpp b/tests/test-thread-safety.cpp index d0b5946e2..5b665b426 100644 --- a/tests/test-thread-safety.cpp diff --git a/llama/patches/0012-model-guard-zero-split-sum-and-name-the-device-index.patch b/llama/patches/0012-model-guard-zero-split-sum-and-name-the-device-index.patch index b9e33e1f7..ab20e8661 100644 --- a/llama/patches/0012-model-guard-zero-split-sum-and-name-the-device-index.patch +++ b/llama/patches/0012-model-guard-zero-split-sum-and-name-the-device-index.patch @@ -1,3 +1,20 @@ +model: fall back to an even split when no device reports free memory, and name the device index +on a bad lookup (the two load_tensors() blocks lifted into free functions so they can be tested). + +STATUS 2026-10-14: FIXED UPSTREAM -- DROP THIS PATCH AT THE FIRST llama.cpp BUMP PAST b11612. +ggml-org/llama.cpp#29677, commit df18bee11 ("model: split layers evenly when no device reports +free memory"), https://github.com/ggml-org/llama.cpp/commit/df18bee11e96557b07aa7b12db0976f3c7ac9165, +puts the same even-split fallback into load_tensors(): + + splits[i] = split_sum != 0.0f ? splits[i] / split_sum : float(i + 1) / n_devices(); + +It is NOT in b11612 (10da8c463, 15 commits earlier on master), so the patch is still needed at that +pin and applies unchanged. Drop-check: `git grep -n split_sum src/llama-model.cpp` at the new tag -- +the guarded line means DROP, not refresh (the patch keeps applying next to it, so the applier will +not notice). Upstream ships no test. The runnable guard src/test/cpp/test_model_split.cpp links the +two functions this patch adds and cannot outlive it as it is; what replaces it is decided in +TODO.md, "Drop patch 0012 at the first bump past b11612". + diff --git a/src/llama-model.cpp b/src/llama-model.cpp index 6872589fa..a56e4bb04 100644 --- a/src/llama-model.cpp diff --git a/llama/patches/0014-common-log-callback-sink.patch b/llama/patches/0014-common-log-callback-sink.patch index 8d1d4aa7a..06d535284 100644 --- a/llama/patches/0014-common-log-callback-sink.patch +++ b/llama/patches/0014-common-log-callback-sink.patch @@ -17,10 +17,10 @@ Upstream-submittable ("common : add a callback sink to common_log for embedding hosts"); not yet filed. Guarded by src/test/cpp/test_common_log_callback.cpp. diff --git a/common/log.cpp b/common/log.cpp -index 86c98e8e6..79c8d8a2a 100644 +index 25f12684e..1e0b89f1e 100644 --- a/common/log.cpp +++ b/common/log.cpp -@@ -168,6 +168,9 @@ struct common_log { +@@ -169,6 +169,9 @@ struct common_log { running = false; t_start = t_us(); @@ -30,31 +30,29 @@ index 86c98e8e6..79c8d8a2a 100644 queue.resize(capacity, common_log_entry(256)); head = 0; tail = 0; -@@ -190,6 +193,9 @@ private: +@@ -191,6 +194,9 @@ private: FILE * file; + common_log_callback callback; + void * callback_user_data; + + bool stdo; bool colors; bool prefix; - bool timestamps; -@@ -205,7 +211,12 @@ private: +@@ -207,7 +213,10 @@ private: bool print_entry(const common_log_entry & e) const { if (e.is_end) return true; -- e.print(); +- if (stdo) { + if (callback) { + // the sink replaces the console, not the file + callback(e.level, e.msg.data(), callback_user_data); -+ } else { -+ e.print(); -+ } - if (file) { - e.print(file); ++ } else if (stdo) { + e.print(); } -@@ -400,6 +411,17 @@ public: + if (file) { +@@ -412,6 +421,17 @@ public: resume(); } @@ -72,8 +70,8 @@ index 86c98e8e6..79c8d8a2a 100644 bool get_colors() const { return colors; } -@@ -497,6 +519,10 @@ void common_log_set_file(struct common_log * log, const char * file) { - log->set_file(file); +@@ -513,6 +533,10 @@ void common_log_set_stdo(struct common_log * log, bool enabled) { + log->set_stdo(enabled); } +void common_log_set_callback(struct common_log * log, common_log_callback callback, void * user_data) { @@ -84,10 +82,10 @@ index 86c98e8e6..79c8d8a2a 100644 if (colors == LOG_COLORS_AUTO) { log->set_colors(tty_can_use_colors()); diff --git a/common/log.h b/common/log.h -index e9e1f1761..5d6ede9aa 100644 +index 93ea58e76..c2353173f 100644 --- a/common/log.h +++ b/common/log.h -@@ -98,6 +98,18 @@ void common_log_set_prefix (struct common_log * log, bool prefix); // w +@@ -99,6 +99,18 @@ void common_log_set_prefix (struct common_log * log, bool prefix); // w void common_log_set_timestamps(struct common_log * log, bool timestamps); // whether to output timestamps in the prefix void common_log_flush (struct common_log * log); // flush all pending log messages diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index f359277cc..7ed99d7ae 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -9,13 +9,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b11574"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b11612"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b11574-"} — call + * plus the resolved upstream commit, e.g. {@code "b11612-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -23,14 +23,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b11574"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b11612"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b11574"; + public static final String LLAMA_CPP_VERSION = "b11612"; // Constants holder — not instantiable. private LlamaCppVersion() {} diff --git a/llama/src/test/cpp/test_common_log_callback.cpp b/llama/src/test/cpp/test_common_log_callback.cpp index 08471b139..dab72e2e3 100644 --- a/llama/src/test/cpp/test_common_log_callback.cpp +++ b/llama/src/test/cpp/test_common_log_callback.cpp @@ -170,3 +170,23 @@ TEST(CommonLogCallback, LevelsPassThroughUnchanged) { common_log_free(log); } + +TEST(CommonLogCallback, SinkTakesPrecedenceOverTheStdoSwitch) { + // b11612 (#30023) added common_log_set_stdo(), a console on/off switch that print_entry() + // consults instead of printing unconditionally. The sink replaces the console, so it is not + // gated on that switch: entries still reach the callback with stdout/stderr output turned off. + recorder rec; + common_log *log = common_log_init(); + common_log_set_stdo(log, false); + common_log_set_callback(log, record, &rec); + + common_log_add(log, GGML_LOG_LEVEL_WARN, "still delivered\n"); + common_log_flush(log); + + const entries got = snapshot(rec); + ASSERT_EQ(got.size(), 1u); + EXPECT_EQ(got[0].first, GGML_LOG_LEVEL_WARN); + EXPECT_EQ(got[0].second, "still delivered\n"); + + common_log_free(log); +} diff --git a/llama/src/test/cpp/test_model_split.cpp b/llama/src/test/cpp/test_model_split.cpp index 18042b3fa..fcfddf0e5 100644 --- a/llama/src/test/cpp/test_model_split.cpp +++ b/llama/src/test/cpp/test_model_split.cpp @@ -13,6 +13,13 @@ // Note that the failure it guards against is NOT reproducible on this side: it needs a GPU // backend that reports zero free memory, and without one `devices` is empty and the mapping is // never reached. What is testable -- and is what actually broke -- is the arithmetic itself. +// +// Upstream fixed the defect itself on 2026-10-14 (ggml-org/llama.cpp#29677, commit df18bee11, +// https://github.com/ggml-org/llama.cpp/commit/df18bee11e96557b07aa7b12db0976f3c7ac9165): the same +// even-split fallback, inline in load_tensors(), and no test. The commit is not in b11612, the pin at +// the time of writing, so the patch and this guard still stand there. At the first bump past it +// patches/0012 is dropped, and with it the two functions this file links -- see TODO.md, "Drop patch +// 0012 at the first bump past b11612", for what replaces this guard. #include "llama-model.h"