From 00657b7950c9f1bd92c505484e3f479a735dd73d Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 14 Oct 2026 23:35:39 +0000 Subject: [PATCH 1/2] Carry patches/0020: validate the leafs before ggml-alloc reuses an allocation The first full Publish run since b11558 (38648257387, on the b11639 merge) crashed ErrorHandlingTest in all three Java test jobs: SIGSEGV in ggml_gallocr_alloc_graph on Linux, 0xC0000005 with both Windows toolchains. The test loads its model with enableEmbedding() and posts an infill request to it. Cause, upstream since #30445 (cb55ffe6e, b11574..b11612): the LM head is expanded by build_logits() after build_pooling(), so an embeddings context reserves a graph expanding t_embd then t_logits, while a generation step on it expands only t_logits, whose first source is output.weight. Same node and leaf counts, the leafs shifted by one. ggml_gallocr_needs_realloc() validates nodes and their sources but of the leafs only the count, so the input leaf out_ids received the slot of a weight (buffer_id -1) and ggml_vbuffer_tensor_alloc() indexed galloc->buffers[-1]. Reproduced without a downloaded model (random 2-layer llama GGUF with FIM tokens, context 128): infill crashes with --embedding and works without; reverting cb55ffe6e passes, and so does validating the leafs like the nodes. With the patch greedy infill on the --embedding model returns the same tokens as without. Guard: test_gallocr_leafs.cpp, two graphs with the same counts and the leafs swapped. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01DdW3CyYtKkK6BQ3VGARo2E --- CLAUDE.md | 4 +- docs/history/llama-cpp-breaking-changes.md | 1 + llama/CMakeLists.txt | 1 + ...e-leafs-before-reusing-an-allocation.patch | 64 +++++++++++ llama/src/test/cpp/test_gallocr_leafs.cpp | 106 ++++++++++++++++++ 5 files changed, 175 insertions(+), 1 deletion(-) create mode 100644 llama/patches/0020-ggml-alloc-validate-leafs-before-reusing-an-allocation.patch create mode 100644 llama/src/test/cpp/test_gallocr_leafs.cpp diff --git a/CLAUDE.md b/CLAUDE.md index 30afdfd1..3b092fe1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1387,6 +1387,7 @@ Current patches: | `0016-model-kolibri1.patch` | **Adds Aleph Alpha's Kolibri-1 (architecture `kolibri1`) ahead of upstream** -- a 78B German/English reasoning MoE (3.46B active) that upstream does not support yet ([ggml-org/llama.cpp#29922](https://github.com/ggml-org/llama.cpp/issues/29922), no PR at the time). A **temporary carry, not a fix**: drop it the moment upstream registers the architecture (`git grep -n kolibri src/llama-arch.cpp` at the new tag), and keep `test_kolibri1.cpp` -- its numerical comparisons must then pass against upstream's implementation, while its GGUF-format rows follow whatever format upstream's converter fixes (see the test-table row). **Architecture** (Aleph Alpha's vLLM reference, `aleph_alpha_inference/kolibri1.py` at `049a6a7`): GQA with per-head q/k RMSNorm; `layer_types` interleaves sliding-window layers (NEOX RoPE) with full-attention layers that have **no** positional encoding; sandwich norms around attention and MoE; every layer routed MoE + one ungated shared expert; the router selects top-k on **`logits + expert_bias`** and weights by the **unbiased `sigmoid(logits)`** -- not DeepSeek-V3's sigmoid router (selection on `sigmoid(logits) + bias`), which picks other experts as soon as the bias is non-zero. The graph therefore builds the selection itself and passes it to `build_moe_ffn` (`selected_experts_in`), leaving upstream's shared MoE code untouched. **Derived from two community ports, which are incompatible with each other's GGUFs** -- Eliasfpv28's AFMoE-based one (base b11378, `expert_gating_func` 2, pre-tokenizer `qwen2`) and the Qwen3-MoE-based one behind Hob-forge's GGUFs (base b11381, as carried in [mjwsolo/localcode#101](https://github.com/mjwsolo/localcode/pull/101): gating 5, pre-tokenizer `kolibri1`, rejects gating 2). This patch **loads both dialects**: gating 2, 5 or absent (all mean the one router), `kolibri1` mapped to the Qwen2 pre-tokenizer in `llama-vocab.cpp`, `expert_weights_norm` / `expert_shared_count` / `output.weight` optional. Every source, with license, is listed in the patch header; `REUSE.toml` annotates the file `MIT AND Apache-2.0` (Eliasfpv28's additions are Apache-2.0). Touches `src/llama-arch.{h,cpp}`, `src/llama-model.cpp` (mapping + NEOX rope list), `src/llama-vocab.cpp` (one name), `src/models/models.h` and the new `src/models/kolibri1.cpp`; no converter. **Runnable guard: `src/test/cpp/test_kolibri1.cpp`** (writes tiny random GGUFs in both dialects and compares every logit with an independent double-precision reference). **Verified against the real 78B model on 2026-10-11** (rented 16-vCPU / 123 GB instance, both published dialects fetched on the machine): both load, and a chat turn through the model's own template answers correctly in each -- `content` `"Berlin."` for a German question and `"51"` for 17x3, with the chain of thought in `reasoning_content` and `finish_reason` `stop`. **The two dialects tokenize identically** (`[452 22090 493 1678 2459 5513 46]`, round-tripping to the input sentence), which is the direct evidence that mapping `kolibri1` to the Qwen2 pre-tokenizer is right. Q4_K_M (44 GB) gave 13.4 t/s and Q3_K_S (31 GB) 14.6 t/s of generation on 16 cores. **Two traps for whoever repeats this:** a *raw* completion on this instruct model degenerates into repetition whatever the patch does, so measure through the chat endpoint; and `chatCompleteText` returns only `content`, which is **empty** when a reasoning model spends its token budget inside the thinking block -- use `handleChatCompletions` and a budget in the hundreds. **Still not verified:** GPU backends -- the community ports report real-model checks (top-1 67/67 against a float64 reference for Q8_0, tool calls, German text). | | `0018-ggml-amx-free-the-aligned-buffer.patch` | **The AMX weight buffer is freed with `free()` but allocated with `_aligned_malloc()` on Windows -- `STATUS_HEAP_CORRUPTION` (0xC0000374) the moment a model is unloaded.** `ggml_backend_amx_buffer_type_alloc_buffer()` (`ggml/src/ggml-cpu/amx/amx.cpp`) allocates with `ggml_aligned_malloc()`, which is `_aligned_malloc()` under `_MSC_VER` (plain clang on a `*-windows-msvc` target defines it too); `ggml_backend_amx_buffer_free_buffer()` called `free()` on it. Fine on Linux (`posix_memalign` + `free`), fatal on Windows, and only there. The patch frees with `ggml_aligned_free(buffer->context, buffer->size)`, exactly what `ggml_backend_cpu_buffer_free_buffer()` does; on Linux that is `free()`, so nothing changes there. **Reached by** the `sapphirerapids` CPU module (the only variant built with `AMX_TILE AMX_INT8`, which `ggml_backend_load_best` picks on Sapphire/Emerald/Granite Rapids; `ggml_amx_init()` is `return true` on `_WIN32`) and a model whose weights have AMX kernels (Q4_0, Q4_1, Q8_0, Q4_K, Q5_K, Q6_K, IQ4_XS -- i.e. every CI model). **Found by the second full run of the modular natives** (38069595101): `Java Tests Windows 2025 x86_64 (clang)` landed on an `Intel(R) Xeon(R) 6973P-C` runner (Granite Rapids), loaded `ggml-cpu-sapphirerapids.dll`, ran `GrammarExample`'s two generations fine and died with exit code -1073740940 at `LlamaModel.close()`; the first run (38065017595) had the same job green on an AMD EPYC 7763 (no AVX-512, `haswell`). **The Windows runners are a CPU lottery**, so which CPU module a Windows job exercises differs per run; before the modular natives the Windows jar was a static Haswell-level build and this code never ran. **Of the six callers of `ggml_aligned_malloc` in the tree this was the only mismatched pair** (`ggml-backend.cpp`, `ggml-cpu.c` twice, `ggml-openvino.cpp`, `ggml-zdnn.cpp` and `ggml.c` all pair correctly), and `ggml-backend.cpp:2464` is the line the fix copies. **Reproduced deliberately rather than waiting for the lottery again**: on a rented Granite Rapids machine, one CPU module staged at a time with the same model and the DLLs CI itself built, `sapphirerapids` crashed **6 of 6** runs -- including at **one** thread, which rules out a data race -- while `cooperlake`, `icelake` and `haswell` were clean 6 of 6, and on Linux with the same CPU all six exit 0 with byte-identical output. The crash lands *after* the inference prints its timings, in cleanup: the signature of a heap corrupted earlier and detected at `free`. With the patch applied the same matrix on the same machine is **clean in all twelve runs**. **The guard is a configure-time check, not a `jllama_test` case**: the AMX code exists only in that one variant of a `GGML_CPU_ALL_VARIANTS` build and executes only on AMX hardware, which the static test build, the local agent's Zen 3 and the cloud sandboxes lack -- so **`llama/cmake/check-aligned-free-pairing.cmake`** fails the configure on **every** platform when a ggml source calls `ggml_aligned_malloc` and never `ggml_aligned_free` (verified red against the unpatched source, naming `amx.cpp`, and green with this patch). A run-time test cannot replace it: without an AMX CPU there is nothing to observe, which is exactly how this shipped. What exercises it: the Windows x86-64 Java test job and the `windows-x86-64` smoke row whenever GitHub hands them an Intel AMX runner; the patch itself is held by `check-patches.py`, applies at b11639, and the patched file compiles with the variant's flags (GCC 13, `-mamx-tile -mamx-int8 -mavx512vnni ...`). Upstream master still carried the `free()` on 2026-10-10; **not yet filed upstream** (nothing before the release). **Drop-check on every bump, note the inversion:** `grep -n "free(buffer->context)" ggml/src/ggml-cpu/amx/amx.cpp` at the new tag -- a **hit** means the bare `free()` is still there and the patch is still needed; **gone** means upstream fixed it, drop the patch. Measured at b11639 (`a0e6dc738`): one hit, line 46. | | `0019-ggml-cpu-select-the-neon-fp32-to-fp16-conversion.patch` | **Every ARM build converts fp32 to fp16 in software, ours and upstream's.** The `__ARM_NEON` branch of `simd-mappings.h` defines both `GGML_CPU_COMPUTE_*` macros and selects only `FP16_TO_FP32`, so `GGML_CPU_FP32_TO_FP16` falls through to `GGML_COMPUTE_FP32_TO_FP16` -- ggml-base's deliberately portable software bit manipulation -- while `neon_compute_fp32_to_fp16`, one `fcvt`, stays defined and referenced by nothing. The `GGML_CPU_*` pair exists to be the *accelerated* variant; on ARM it was defined to be the unaccelerated one. `__POWER9_VECTOR__` and `__riscv_zfhmin` select both, and the comment on that fallback has always promised NEON does too. **Not a regression** -- absent in all 25 most recent commits touching the file, back to 2025-04-10. **Why only ARM feels it:** `ggml_cpu_fp32_to_fp16`, the `from_float` of `GGML_TYPE_F16`, has vector paths for F16C, RISC-V and s390x (the last from #30157) and **none** for NEON, so there the whole function is the scalar loop through the macro. **In the shipped artifacts:** no `fcvt` of any form in that function in any of this project's 17 ARM libraries (`cpu-linux-aarch64` 8 variants, `cpu-android-aarch64` 7, `cpu-windows-aarch64`, `metal-macos-aarch64`) nor in upstream's own `bin-ubuntu-arm64` / `bin-macos-arm64`; a plain `gcc` with no flags defines `__ARM_NEON` and `__ARM_FP16_FORMAT_IEEE` on aarch64, so nothing can miss it. **Measured 2.45x-13.3x** on five ARM core types, three operating systems and nine compiler builds, with the opposite direction in the same run as a control (within 3.2 % everywhere); **bit-exact over all 2^32 float inputs**, differing only in NaN payload, where x86 F16C gives the same numbers -- so it makes ARM agree with what x86 already ships. **Not an inference speed-up:** `perf` puts 83.9 % of an F16 `pp512` run in tinyBLAS's gemm, which converts in its own microkernel, and 0.03 % here. The patch also replaces the stale comment above the fallback with an `#error` pair that refuses a branch defining a COMPUTE macro without selecting it -- its position chosen by a mutation measurement, not by taste (4 of 4 correctness cases caught before the fallback against 3 of 4 directly after the chain). **The `FP16_TO_FP32` half carries `&& !defined(__F16C__)`, and that carve-out is load-bearing since the b11639 bump:** upstream #30343 (`685f70894` = b11635, merged 2026-10-14) puts that one direction back on the lookup table deliberately -- clang lowers the F16C intrinsic into the live accumulator register of the quantized dot products and serialises the loop, 45-65 % slower Q8_0/Q4_0 `MUL_MAT_ID` -- so without it the guard fires on **11 of the 14 x86-64 CPU variant modules**. Up to b11632 #30343 was absent, the `__F16C__` block still selected both directions and the carve-out was inert; from b11639 on the pin carries #30343, so x86 FP16->FP32 is on the lookup table exactly as upstream ships it, and the carve-out is what keeps the guard quiet there (`gcc -E` of the patched b11639 header silent for `-march=x86-64`, `-msse4.2`, `-march=haswell`, `-march=sapphirerapids`, `-march=znver4` and `-mavx -mf16c`). A **first** version of this patch also moved #30270's two F16C selections into the `__F16C__` branch; that was dropped when #30343 turned the block into a documented exception with an issue link, which is also why this diff now applies **both** at the pin and on upstream master (verified with `git apply --check` against each, and `clang -E` silent in all five configurations of the patched pinned source). Full record: [`docs/benchmarks/arm-fp16-conversion.md`](docs/benchmarks/arm-fp16-conversion.md). **Submitted upstream as [ggml-org/llama.cpp#30505](https://github.com/ggml-org/llama.cpp/pull/30505)** (open, rebased onto master 2026-10-14 at the maintainer's request), byte-identical to this diff -- the diff body of the patch file and `git diff` of the PR commit hash the same. **Drop-check on every bump, and its output must be read rather than tested:** `git show :ggml/src/ggml-cpu/simd-mappings.h \| sed -n '/^#if defined(__ARM_NEON)/,/^#elif/p' \| grep -n GGML_CPU_FP32_TO_FP16` prints **nothing** while the patch is needed and one line once upstream merged it -- then **drop this patch rather than refreshing it**. **Do not reduce the looser `git grep -n "GGML_CPU_FP32_TO_FP16(x) GGML_CPU_COMPUTE_FP32_TO_FP16"` over that file to a true/false:** it returns **three** hits at every tag so far (`__POWER9_VECTOR__`, `__riscv_zfhmin` and the out-of-chain `__F16C__` block, the first two with the NEON line's own four-space indent), so a `grep -q` reports "upstream fixed it" at every tag and deletes a fix that is still needed -- measured at b11639 (`a0e6dc738`): lines 71, 114 and 150, none in the `__ARM_NEON` branch, while `git apply --check` of the patch is clean. | +| `0020-ggml-alloc-validate-leafs-before-reusing-an-allocation.patch` | **An `--embedding` model that is asked to generate crashes the process in `ggml_gallocr_alloc_graph`** (SIGSEGV on Linux, 0xC0000005 with both Windows toolchains), since upstream #30445 (`cb55ffe6e`, "graph: skip the LM head for pooled embeddings", b11574..b11612). `ggml_gallocr_alloc_graph()` assigns every tensor by **position** from the graph it was reserved with, and `ggml_gallocr_needs_realloc()` validates every node and every node's sources but of the **leafs only the count**. Since #30445 the LM head is expanded by `build_logits()` *after* `build_pooling()`, so an embeddings context reserves a graph that expands `t_embd` first and `t_logits` second, while a generation step on it (the server calls `llama_set_embeddings(ctx, false)` for a slot that samples) expands only `t_logits`, whose first source is `output.weight`: same node and leaf counts, every leaf after it shifted by one. An input leaf then lands on the slot of a weight (`buffer_id -1`) and, with the asserts compiled out, `ggml_vbuffer_tensor_alloc()` indexes `galloc->buffers[-1]`; a leaf landing on another input's slot is not size-checked either. The patch validates the leafs exactly as the nodes are validated, so an invalid leaf makes the allocator reserve again (one buffer) or return false, on which `ggml_backend_sched` reserves the graph itself. **Found by** Publish run 38648257387 (2026-10-14, the first full run since b11558): `ErrorHandlingTest` loads its model with `enableEmbedding()` and posts an infill request to it -- its first test in JUnit's order -- and the fork died in all three Java test jobs (`ggml_gallocr_alloc_graph <- ggml_backend_sched_alloc_graph <- llama_context::process_ubatch <- decode <- server_context_impl::update_slots`). **Reproduced locally without a downloaded model** (a random 2-layer llama GGUF with FIM tokens, context 128): infill crashes with `--embedding` and works without; a debug print showed leaf 31 `out_ids` arriving at a slot with no allocation; reverting `cb55ffe6e` makes it pass, and so does this patch -- with greedy sampling the `--embedding` model then returns the same tokens as the model without, run after run. Upstream master (`a0e6dc738`) still lacks the check; **not yet filed upstream**. **Runnable guard: `src/test/cpp/test_gallocr_leafs.cpp`** (two graphs with the same counts and the leafs swapped). **Drop-check on every bump:** read `ggml_gallocr_needs_realloc()` in `ggml/src/ggml-alloc.c` at the new tag -- once it walks `graph->leafs` itself, drop this patch rather than refreshing it. | **Dropped patches** (`0004`, `0005`, `0009`, `0010`, `0011`, `0012`, `0013` -- upstream fixed each defect itself) are recorded in [`docs/history/dropped-llama-patches.md`](docs/history/dropped-llama-patches.md): @@ -2432,8 +2433,9 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" | `src/test/cpp/test_rpc.cpp` | 28 | **The runnable guard for `patches/0015` and for `rpc_support.hpp`.** The device-selection rules (pure, literal inputs: `--rpc` endpoints accumulate across repeated options, an explicit `--device`/`-dev` is never overridden, stale RPC devices are replaced by the local GPUs chosen the way llama.cpp's default does — duplicates by device id dropped, iGPUs only without a discrete GPU, `none` on a CPU-only host). Then the **real** ggml-rpc client and server over loopback, no model: a `mul_mat` graph computed on the RPC backend equals the local CPU result; `ggml_backend_rpc_stop_server()` ends the server, frees the port and disconnects a client that is still connected; an unreachable or malformed endpoint is `nullptr`/`std::invalid_argument` instead of an abort; a registered server that went away is re-checked on the next registration; its device reports 0/0 memory instead of aborting; and a stale server is kept out of a later load that did not ask for it. The loopback server picks a free port from a range so parallel jobs do not collide, and serves the **CPU** by name — a served GPU that lacks an op aborts the server (see the RPC section). Also: server devices chosen by name (case-insensitive, deduplicated), an unknown name listing the available devices, and a registered RPC device refused; the mmproj device pinned the way clip would choose it minus the stale server; and `exclude_stale_devices()`, the params-level guard for TTS and the trainer. | | `src/test/cpp/test_native_server_shutdown.cpp` | 3 | **The runnable guard for the shutdown-handler guard in `patches/0006`/`0007`.** Runs the real `llama_server()` in router mode over an empty `--models-dir` on an ephemeral loopback port (no model, no worker) and stops it the way `native_server.cpp` does. Pins a clean stop (exit code 0), that `llama_server_request_shutdown()` **after** the server returned is a no-op (the deterministic form of the CI `SIGSEGV` — it crashed every run before the fix), and that requests hammered from another thread while the server tears down are safe (5 rounds). Compiled on non-Android only, like `server.cpp` itself. | +| `src/test/cpp/test_gallocr_leafs.cpp` | 1 | **The runnable guard for `patches/0020`.** Two graphs with one node and two leafs each, the leafs swapped (`add(w, x)` visits the weight first, `add(x, w)` the input first): the allocator is reserved with the first, the second must still get an allocation for the input inside the allocator's buffer and compute the right sum on it. No model, every platform; before the patch the allocation step indexed `galloc->buffers[-1]`. | -**Current total: 601 tests (all passing).** +**Current total: 602 tests (all passing).** #### Upstream source location (in CMake build tree) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 10bcb440..7fdd3044 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -894,3 +894,4 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b11617–b11632 | patches + upstream verification | **`0012` dropped, twelve patches apply unchanged** (`git apply` in order on pristine `df18bee11` and `62df1815b` worktrees, 54 modified files; `check-patches.py`: 12 patches, 95 hunks, 0 problems). **The drop-check fired both ways:** `src/llama-model.cpp:1650` is the guarded `splits[i] = split_sum != 0.0f ? ... : float(i + 1) / n_devices()` now, and `git apply --check` of `0012` on the `df18bee11` tree fails at `src/llama-model.cpp:1638` -- upstream's line is inside the block the patch replaced, so unlike `0010` this one could not have been carried on by mistake. **The first bump pinned ahead of the tag**: no tag past b11618 existed on 2026-10-14, so `JLLAMA_GIT_TAG b11632` is paired with `JLLAMA_GIT_COMMIT 62df1815b…` and the production path ran exactly as CI will -- `FetchContent` by commit, the applier (stamp head `62df1815b`), and the pin check reporting `fetched commit count 11632 = b11632` (`git rev-list --count`, the number upstream's `release.yml` stamps the tag with and the one llama.cpp's own `build-info` reports). Then the static test build, `ctest` **601/601** (608 minus the seven of `test_model_split.cpp`), and the model-free native Java tests over the library (`NativeLibraryLoadSmokeTest` with the `LlamaCppVersion` pin `b11632` against the linked `b11632-62df1815b`, `LlamaLoggerTest`, `RpcServerTest`, `BackendLoadTest`: 27/27). The same chain had run at `df18bee11` / b11626 first (601/601, 27/27, `b11626-df18bee11`). The other drop-checks against pristine `62df1815b`: no `kolibri` (`0016`), no `common_log` hook (`0014`), no `stop_server` (`0015`), uncast `_mm_prefetch` (`0017`), `free(buffer->context)` (`0018`), the software fp16 conversion on `__ARM_NEON` (`0019`), the `GetCommandLineW` override and no `common_params_parse_main` (`0001`), the unconditional load-progress assignment (`0002`), no `LLAMA_SERVER_WORKER_CMD` (`0008`), no `get_slot_prompt_similarity` (`0003`). `check-natives.py`: 23 natives jars, 0 disagreements; spotless clean. Not run locally: the model-backed Java suite and the modular build -- CI covers both. When upstream publishes `b11632`, clear `JLLAMA_GIT_COMMIT` (`llama-next-version.sh` prints the NOTE); nothing else moves. | | b11632–b11639 | 7 commits on master's head of 2026-10-14 (`a0e6dc738` = b11639, pinned through `JLLAMA_GIT_COMMIT` ahead of the tag -- upstream's newest tag is b11627), 21 files, ~691/90 lines (57 KiB; no `tools/ui`, no `.github/workflows`, no `common/`, `include/`, `src/`, `tools/`, root `CMakeLists.txt` unchanged, so the `_DEFAULT` re-check of "Modular natives" has nothing new). `ggml/src/ggml-cpu/ggml-cpu.c` (#30454, b11633): `GGML_OP_SET_ROWS` moves to the `n_tasks = n_threads` group -- every thread ran it with its own row of the F16 -> F32 conversion buffer while the planner sized the work buffer for one, so threads > 0 wrote past it whenever the graph held another multi-threaded op; every CPU module rebuilds it. `ggml/src/ggml-cpu/simd-mappings.h` (#30343, b11635): x86 with F16C keeps the lookup table for FP16 -> FP32 (clang lowers the F16C intrinsic into the dot products' live accumulator and serialises the loop, 45-65 % slower Q8_0/Q4_0 `MUL_MAT_ID`; the Windows x86-64 jar is built with plain clang) -- the carve-out of `0019`, see the patches row. `ggml-sycl` (#29107, b11636): persistent IQ3_S/IQ3_XXS reorder **and a build change: `GGML_SYCL_DNN` (default ON) now fails the configure when oneDNN is missing or CPU-only** instead of building without it -- `build-linux-x86_64-sycl` installed only DPC++ + MKL (the shipped `libggml-sycl.so` of the b11558 snapshot has no `libdnnl` in its `DT_NEEDED`), so the job now also installs `intel-oneapi-dnnl-devel`; the Windows job already installed `intel.oneapi.win.dnnl` and its module imports `dnnl.dll`. `ggml-opencl` (#30186, b11639): a strided CLAMP kernel for non-contiguous views (the OpenCL module jars). Docs, `gguf-py` and `.devops` Dockerfiles otherwise. **No project source change; one workflow change (the oneDNN package).** | | b11632–b11639 | patches + upstream verification | **All twelve patches apply unchanged** (`git apply` in order on a pristine `a0e6dc738` worktree; `check-patches.py`: 12 patches, 94 hunks, 0 problems). **`0019`'s `&& !defined(__F16C__)` carve-out is load-bearing from this pin on**, and holds: `gcc -E` of the patched `simd-mappings.h` is silent for `-march=x86-64`, `-msse4.2`, `-march=haswell`, `-march=sapphirerapids`, `-march=znver4` and `-mavx -mf16c`. Drop-checks against pristine b11639 are those of the b11632 row, the range touching no other patch-target file: no `kolibri` (`0016`), no `common_log` hook (`0014`), no `stop_server` (`0015`), uncast `_mm_prefetch` (`0017`), `free(buffer->context)` (`0018`), the software fp32 -> fp16 conversion on `__ARM_NEON` (`0019`, #30505 still open), the `GetCommandLineW` override and no `common_params_parse_main` (`0001`), the unconditional load-progress assignment (`0002`), no `LLAMA_SERVER_WORKER_CMD` (`0008`), no `get_slot_prompt_similarity` (`0003`). Through the production path -- `rm -rf build`, `FetchContent` by commit, the applier, the pin check reporting `fetched commit count 11639 = b11639` -- the static test build, `ctest` **601/601**, and the model-free native Java tests over the library (`NativeLibraryLoadSmokeTest` with the `LlamaCppVersion` pin `b11639` against the linked build, `LlamaLoggerTest`, `RpcServerTest`, `BackendLoadTest`: 27/27). `check-natives.py`: 23 natives jars, 0 disagreements. Not run locally: the model-backed Java suite, the modular build and the SYCL job with oneDNN -- CI covers them. | +| b11639 (follow-up) | `patches/0020`, a runtime regression from b11574..b11612 | **An `--embedding` context that is asked to generate crashed the process** -- found only now because no Publish run reached the Java tests between b11558 and the b11639 dispatch (38648257387): `ErrorHandlingTest` (model with `enableEmbedding()`, first test an infill request) died with SIGSEGV in `ggml_gallocr_alloc_graph` on Linux and 0xC0000005 under both Windows toolchains. Cause: #30445 (`cb55ffe6e`, "graph: skip the LM head for pooled embeddings") moved the LM head's expansion into `build_logits()` after `build_pooling()`, so the reserved embeddings graph and a generation graph on the same context have the same node and leaf counts but the leafs in another order, and `ggml_gallocr_needs_realloc()` validates nodes and their sources but of the leafs only the count -- an input leaf (`out_ids`) received the slot of a weight and `ggml_vbuffer_tensor_alloc()` indexed `galloc->buffers[-1]`. Reproduced locally with a random 2-layer llama GGUF (FIM tokens set), confirmed by reverting `cb55ffe6e`, fixed by `0020` (validate the leafs like the nodes), greedy output identical with and without `--embedding`; guard `test_gallocr_leafs.cpp`. Upstream master (`a0e6dc738`) unchanged. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index 0e1e1b58..8ed74345 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -1104,6 +1104,7 @@ if(BUILD_TESTING) src/test/cpp/test_model_flags.cpp src/test/cpp/test_wire_contracts.cpp src/test/cpp/test_rpc.cpp + src/test/cpp/test_gallocr_leafs.cpp ${llama.cpp_SOURCE_DIR}/tools/server/server-common.cpp ${llama.cpp_SOURCE_DIR}/tools/server/server-chat.cpp ${llama.cpp_SOURCE_DIR}/tools/server/server-context.cpp diff --git a/llama/patches/0020-ggml-alloc-validate-leafs-before-reusing-an-allocation.patch b/llama/patches/0020-ggml-alloc-validate-leafs-before-reusing-an-allocation.patch new file mode 100644 index 00000000..eba49312 --- /dev/null +++ b/llama/patches/0020-ggml-alloc-validate-leafs-before-reusing-an-allocation.patch @@ -0,0 +1,64 @@ +ggml-alloc : validate the leafs before reusing a graph's allocation + +ggml_gallocr_alloc_graph() assigns every tensor of a graph by position from the previous +ggml_gallocr_reserve(): node i takes node_allocs[i], leaf i takes leaf_allocs[i]. +ggml_gallocr_needs_realloc() decides whether those assignments still fit, and it validates every +node and every node's sources -- but of the leafs it compares only the count. A graph with the +same number of leafs in another order therefore reuses the old leaf assignments: an input leaf +lands on a slot that belonged to a weight (buffer_id -1, nothing allocated), and with NDEBUG the +asserts in ggml_gallocr_init_tensor() are gone, so ggml_vbuffer_tensor_alloc() indexes +galloc->buffers[-1]. A leaf that lands on a slot of another input is not caught either: its size +is never checked, so it may get a region too small for it. + +Reached since #30445 ("graph: skip the LM head for pooled embeddings", cb55ffe6e, b11574..b11612): +the models no longer expand their LM head themselves, build_logits() does, after build_pooling(). +In a context with embeddings enabled the reserved graph expands t_embd first and t_logits +second; a generation step on the same context (the server switches embeddings off for a slot +that samples, llama_set_embeddings(ctx, false)) expands only t_logits, whose first source is +output.weight. Same node and leaf counts, every leaf after output.weight shifted by one. Found +by java-llama.cpp's Publish run 38648257387 (2026-10-14): ErrorHandlingTest loads its model with +--embedding and posts an infill request to it, and the forked JVM died with SIGSEGV in +ggml_gallocr_alloc_graph on Linux, and with 0xC0000005 under both Windows toolchains: + + ggml_gallocr_alloc_graph <- ggml_backend_sched_alloc_graph <- llama_context::process_ubatch + <- llama_context::decode <- server_context_impl::update_slots + +Reproduced locally with a random 2-layer llama GGUF (FIM tokens set) and a context of 128: +infill on the model without --embedding works, with --embedding it crashes; a debug print shows +leaf 31 'out_ids' arriving at a slot with no allocation. Reverting cb55ffe6e makes it pass, and +so does this patch, which validates the leafs exactly as the nodes are validated: an invalid +leaf makes the allocator reserve again (one buffer) or report false, on which +ggml_backend_sched reserves the graph itself. With it, greedy infill on the --embedding model +returns the same tokens as on the model without, run after run. + +The guard is src/test/cpp/test_gallocr_leafs.cpp: two graphs with the same node and leaf counts +and the leafs swapped, reserve with one, allocate the other. + +Drop-check on every bump: read ggml_gallocr_needs_realloc() in ggml/src/ggml-alloc.c at the new +tag -- once it walks graph->leafs itself, drop this patch rather than refreshing it. + +diff --git a/ggml/src/ggml-alloc.c b/ggml/src/ggml-alloc.c +index b0c067b49..35e4d2573 100644 +--- a/ggml/src/ggml-alloc.c ++++ b/ggml/src/ggml-alloc.c +@@ -1021,6 +1021,20 @@ static bool ggml_gallocr_needs_realloc(ggml_gallocr_t galloc, struct ggml_cgraph + return true; + } + ++ // the leafs are allocated by position too, so a graph whose leafs come in another order -- the same ++ // count, e.g. outputs expanded in a different order -- must not reuse the previous assignments ++ for (int i = 0; i < graph->n_leafs; i++) { ++ struct ggml_tensor * leaf = graph->leafs[i]; ++ struct leaf_alloc * leaf_alloc = &galloc->leaf_allocs[i]; ++ ++ if (!ggml_gallocr_node_needs_realloc(galloc, leaf, &leaf_alloc->leaf)) { ++#ifndef NDEBUG ++ GGML_LOG_DEBUG("%s: leaf %s is not valid\n", __func__, leaf->name); ++#endif ++ return true; ++ } ++ } ++ + for (int i = 0; i < graph->n_nodes; i++) { + struct ggml_tensor * node = graph->nodes[i]; + struct node_alloc * node_alloc = &galloc->node_allocs[i]; diff --git a/llama/src/test/cpp/test_gallocr_leafs.cpp b/llama/src/test/cpp/test_gallocr_leafs.cpp new file mode 100644 index 00000000..daa712d8 --- /dev/null +++ b/llama/src/test/cpp/test_gallocr_leafs.cpp @@ -0,0 +1,106 @@ +// SPDX-FileCopyrightText: 2026 Bernard Ladenthin +// +// SPDX-License-Identifier: MIT +// +// Runnable guard for patches/0020 (ggml-alloc: validate the leafs before reusing an allocation). +// +// ggml_gallocr_alloc_graph() hands every tensor the slot of the same position in the graph the +// allocator was reserved with. Before the patch, ggml_gallocr_needs_realloc() validated the nodes +// and their sources but compared only the NUMBER of leafs, so a graph with the same leaf count in +// another order reused the old leaf slots: an input leaf landed on the slot of a weight (no +// allocation, buffer_id -1) and ggml_vbuffer_tensor_alloc() indexed galloc->buffers[-1]. Since +// llama.cpp #30445 that is exactly what an --embedding context does when the server samples on it +// (t_embd + t_logits expanded at reserve, only t_logits at decode); it killed the JVM in +// ErrorHandlingTest on every platform. +// +// The two graphs below have one node and two leafs each, the leafs swapped: add(w, x) visits the +// weight first, add(x, w) the input first. Reserved with the first, the second must still get an +// allocation for x inside the allocator's buffer -- and compute the right sum on it. Without the +// patch the allocation step reads past galloc->buffers and the test dies or fails here, on every +// platform, without a model. + +#include "ggml-alloc.h" +#include "ggml-backend.h" +#include "ggml-cpu.h" +#include "ggml.h" + +#include + +#include +#include +#include + +namespace { + +constexpr int64_t N = 16; + +struct ctx_deleter { + void operator()(ggml_context *ctx) const { ggml_free(ctx); } +}; +using ctx_ptr = std::unique_ptr; + +ctx_ptr new_no_alloc_context(size_t n_tensors, size_t n_graphs) { + ggml_init_params params = { + /*.mem_size =*/ggml_tensor_overhead() * n_tensors + ggml_graph_overhead() * n_graphs, + /*.mem_buffer =*/nullptr, + /*.no_alloc =*/true, + }; + return ctx_ptr(ggml_init(params)); +} + +} // namespace + +TEST(GallocrLeafOrder, AGraphWithItsLeafsInAnotherOrderIsAllocatedAgain) { + ggml_backend_t backend = ggml_backend_cpu_init(); + ASSERT_NE(backend, nullptr); + + // the "weight": a leaf that already has data, so the reserve gives it no slot of its own + ctx_ptr wctx = new_no_alloc_context(1, 0); + ggml_tensor *w = ggml_new_tensor_1d(wctx.get(), GGML_TYPE_F32, N); + ggml_set_name(w, "w"); + ggml_backend_buffer_t wbuf = ggml_backend_alloc_ctx_tensors(wctx.get(), backend); + ASSERT_NE(wbuf, nullptr); + + // the "input": a leaf without data, which the allocator has to place + ctx_ptr gctx = new_no_alloc_context(8, 2); + ggml_tensor *x = ggml_new_tensor_1d(gctx.get(), GGML_TYPE_F32, N); + ggml_set_name(x, "x"); + ggml_set_input(x); + + ggml_cgraph *reserved = ggml_new_graph(gctx.get()); + ggml_build_forward_expand(reserved, ggml_add(gctx.get(), w, x)); // leafs: w, x + ggml_tensor *out = ggml_add(gctx.get(), x, w); + ggml_set_output(out); + ggml_cgraph *decoded = ggml_new_graph(gctx.get()); + ggml_build_forward_expand(decoded, out); // leafs: x, w + ASSERT_EQ(ggml_graph_n_nodes(reserved), ggml_graph_n_nodes(decoded)); + + ggml_gallocr_t galloc = ggml_gallocr_new(ggml_backend_get_default_buffer_type(backend)); + ASSERT_NE(galloc, nullptr); + ASSERT_TRUE(ggml_gallocr_reserve(galloc, reserved)); + ASSERT_TRUE(ggml_gallocr_alloc_graph(galloc, decoded)); + + ASSERT_NE(x->data, nullptr) << "the input leaf got no allocation"; + ASSERT_NE(x->buffer, nullptr) << "the input leaf is not in the allocator's buffer"; + const auto *base = static_cast(ggml_backend_buffer_get_base(x->buffer)); + const auto *data = static_cast(x->data); + ASSERT_GE(data, base); + ASSERT_LE(data + ggml_nbytes(x), base + ggml_backend_buffer_get_size(x->buffer)); + + std::vector wv(N), xv(N), ov(N); + for (int64_t i = 0; i < N; i++) { + wv[i] = static_cast(i); + xv[i] = 100.0f; + } + ggml_backend_tensor_set(w, wv.data(), 0, ggml_nbytes(w)); + ggml_backend_tensor_set(x, xv.data(), 0, ggml_nbytes(x)); + ASSERT_EQ(ggml_backend_graph_compute(backend, decoded), GGML_STATUS_SUCCESS); + ggml_backend_tensor_get(out, ov.data(), 0, ggml_nbytes(out)); + for (int64_t i = 0; i < N; i++) { + EXPECT_FLOAT_EQ(ov[i], 100.0f + static_cast(i)) << "at " << i; + } + + ggml_gallocr_free(galloc); + ggml_backend_buffer_free(wbuf); + ggml_backend_free(backend); +} From 1918d6a7c5e331ab72cf2ba7e9071c8c9e480859 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 14 Oct 2026 23:42:33 +0000 Subject: [PATCH 2/2] Guard patches/0020 with graphs whose nodes match and only the leafs move The first version of test_gallocr_leafs.cpp swapped the operands of one add, which moves a node source too -- the check that existed before the patch already caught that, so the test passed without the fix. The guard now keeps both nodes and their sources in place and moves only the leafs, the shape of the llama.cpp case: it dies with SIGSEGV without the patch and passes with it (602/602). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01DdW3CyYtKkK6BQ3VGARo2E --- CLAUDE.md | 2 +- ...e-leafs-before-reusing-an-allocation.patch | 4 +-- llama/src/test/cpp/test_gallocr_leafs.cpp | 27 ++++++++++++------- 3 files changed, 20 insertions(+), 13 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 3b092fe1..89874e2f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2433,7 +2433,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" | `src/test/cpp/test_rpc.cpp` | 28 | **The runnable guard for `patches/0015` and for `rpc_support.hpp`.** The device-selection rules (pure, literal inputs: `--rpc` endpoints accumulate across repeated options, an explicit `--device`/`-dev` is never overridden, stale RPC devices are replaced by the local GPUs chosen the way llama.cpp's default does — duplicates by device id dropped, iGPUs only without a discrete GPU, `none` on a CPU-only host). Then the **real** ggml-rpc client and server over loopback, no model: a `mul_mat` graph computed on the RPC backend equals the local CPU result; `ggml_backend_rpc_stop_server()` ends the server, frees the port and disconnects a client that is still connected; an unreachable or malformed endpoint is `nullptr`/`std::invalid_argument` instead of an abort; a registered server that went away is re-checked on the next registration; its device reports 0/0 memory instead of aborting; and a stale server is kept out of a later load that did not ask for it. The loopback server picks a free port from a range so parallel jobs do not collide, and serves the **CPU** by name — a served GPU that lacks an op aborts the server (see the RPC section). Also: server devices chosen by name (case-insensitive, deduplicated), an unknown name listing the available devices, and a registered RPC device refused; the mmproj device pinned the way clip would choose it minus the stale server; and `exclude_stale_devices()`, the params-level guard for TTS and the trainer. | | `src/test/cpp/test_native_server_shutdown.cpp` | 3 | **The runnable guard for the shutdown-handler guard in `patches/0006`/`0007`.** Runs the real `llama_server()` in router mode over an empty `--models-dir` on an ephemeral loopback port (no model, no worker) and stops it the way `native_server.cpp` does. Pins a clean stop (exit code 0), that `llama_server_request_shutdown()` **after** the server returned is a no-op (the deterministic form of the CI `SIGSEGV` — it crashed every run before the fix), and that requests hammered from another thread while the server tears down are safe (5 rounds). Compiled on non-Android only, like `server.cpp` itself. | -| `src/test/cpp/test_gallocr_leafs.cpp` | 1 | **The runnable guard for `patches/0020`.** Two graphs with one node and two leafs each, the leafs swapped (`add(w, x)` visits the weight first, `add(x, w)` the input first): the allocator is reserved with the first, the second must still get an allocation for the input inside the allocator's buffer and compute the right sum on it. No model, every platform; before the patch the allocation step indexed `galloc->buffers[-1]`. | +| `src/test/cpp/test_gallocr_leafs.cpp` | 1 | **The runnable guard for `patches/0020`.** Two graphs with the same two nodes in the same order (`n1 = add(x, x)`, `n2 = mul(w, n1)`) and only the leafs moved: one expands `n1` before `n2` and meets the input first, the other expands only `n2`, whose first source is the weight -- the shape of the llama.cpp case, where every node and node source still matches its slot, so the old check passed it. Reserved with the first, the second must get an allocation for the input inside the allocator's buffer and compute the right product on it. No model, every platform. **Verified both ways:** with the fix removed from the source tree the test dies with SIGSEGV (rc 139); a first version that swapped the operands of one `add` passed without the fix, because a swapped *node source* was already caught -- a guard has to move only the leafs. | **Current total: 602 tests (all passing).** diff --git a/llama/patches/0020-ggml-alloc-validate-leafs-before-reusing-an-allocation.patch b/llama/patches/0020-ggml-alloc-validate-leafs-before-reusing-an-allocation.patch index eba49312..6bcb1110 100644 --- a/llama/patches/0020-ggml-alloc-validate-leafs-before-reusing-an-allocation.patch +++ b/llama/patches/0020-ggml-alloc-validate-leafs-before-reusing-an-allocation.patch @@ -31,8 +31,8 @@ leaf makes the allocator reserve again (one buffer) or report false, on which ggml_backend_sched reserves the graph itself. With it, greedy infill on the --embedding model returns the same tokens as on the model without, run after run. -The guard is src/test/cpp/test_gallocr_leafs.cpp: two graphs with the same node and leaf counts -and the leafs swapped, reserve with one, allocate the other. +The guard is src/test/cpp/test_gallocr_leafs.cpp: two graphs with the same nodes in the same order +and only the leafs moved (it dies with SIGSEGV without this patch), reserve with one, allocate the other. Drop-check on every bump: read ggml_gallocr_needs_realloc() in ggml/src/ggml-alloc.c at the new tag -- once it walks graph->leafs itself, drop this patch rather than refreshing it. diff --git a/llama/src/test/cpp/test_gallocr_leafs.cpp b/llama/src/test/cpp/test_gallocr_leafs.cpp index daa712d8..a0305e9d 100644 --- a/llama/src/test/cpp/test_gallocr_leafs.cpp +++ b/llama/src/test/cpp/test_gallocr_leafs.cpp @@ -13,11 +13,15 @@ // (t_embd + t_logits expanded at reserve, only t_logits at decode); it killed the JVM in // ErrorHandlingTest on every platform. // -// The two graphs below have one node and two leafs each, the leafs swapped: add(w, x) visits the -// weight first, add(x, w) the input first. Reserved with the first, the second must still get an -// allocation for x inside the allocator's buffer -- and compute the right sum on it. Without the -// patch the allocation step reads past galloc->buffers and the test dies or fails here, on every -// platform, without a model. +// The two graphs below have the same two nodes in the same order -- n1 = add(x, x), then +// n2 = mul(w, n1) -- so every node and every node source matches its slot, which the old check did +// verify. Only the leafs move: the first graph expands n1 before n2 and meets the input x first, the +// second expands only n2, whose first source is the weight w. That is the shape of the llama.cpp +// case (t_embd then t_logits at reserve, only t_logits at decode, output.weight first). Reserved +// with the first, the second must still get an allocation for x inside the allocator's buffer and +// compute the right product on it. Without the patch x takes the slot of w (no allocation) and the +// allocation step reads galloc->buffers[-1]; the test dies or fails here, on every platform, without +// a model. #include "ggml-alloc.h" #include "ggml-backend.h" @@ -67,12 +71,15 @@ TEST(GallocrLeafOrder, AGraphWithItsLeafsInAnotherOrderIsAllocatedAgain) { ggml_set_name(x, "x"); ggml_set_input(x); - ggml_cgraph *reserved = ggml_new_graph(gctx.get()); - ggml_build_forward_expand(reserved, ggml_add(gctx.get(), w, x)); // leafs: w, x - ggml_tensor *out = ggml_add(gctx.get(), x, w); + ggml_tensor *n1 = ggml_add(gctx.get(), x, x); + ggml_tensor *out = ggml_mul(gctx.get(), w, n1); ggml_set_output(out); + + ggml_cgraph *reserved = ggml_new_graph(gctx.get()); + ggml_build_forward_expand(reserved, n1); // leafs: x, w + ggml_build_forward_expand(reserved, out); // nodes: n1, out ggml_cgraph *decoded = ggml_new_graph(gctx.get()); - ggml_build_forward_expand(decoded, out); // leafs: x, w + ggml_build_forward_expand(decoded, out); // leafs: w, x -- nodes: n1, out ASSERT_EQ(ggml_graph_n_nodes(reserved), ggml_graph_n_nodes(decoded)); ggml_gallocr_t galloc = ggml_gallocr_new(ggml_backend_get_default_buffer_type(backend)); @@ -97,7 +104,7 @@ TEST(GallocrLeafOrder, AGraphWithItsLeafsInAnotherOrderIsAllocatedAgain) { ASSERT_EQ(ggml_backend_graph_compute(backend, decoded), GGML_STATUS_SUCCESS); ggml_backend_tensor_get(out, ov.data(), 0, ggml_nbytes(out)); for (int64_t i = 0; i < N; i++) { - EXPECT_FLOAT_EQ(ov[i], 100.0f + static_cast(i)) << "at " << i; + EXPECT_FLOAT_EQ(ov[i], 200.0f * static_cast(i)) << "at " << i; } ggml_gallocr_free(galloc);