diff --git a/README.md b/README.md index 6b9b7ccc..da20ee41 100644 --- a/README.md +++ b/README.md @@ -287,24 +287,30 @@ platforms you target, e.g. `cpu-linux-x86-64`. > **Two GPU jars need more than their runtime, and both were found by measurement rather than by > reading the code** ([`docs/benchmarks/gpu-module-jars.md`](docs/benchmarks/gpu-module-jars.md)). > -> **CUDA: the driver must be new enough for the toolkit the module was built with (13.4).** The -> module ships native code for the GPU generations listed in `CMAKE_CUDA_ARCHITECTURES` -> (Turing through Blackwell); a GPU outside that set runs only if the driver can compile the -> module's PTX, and **a driver too old to do so does not fall back to the CPU -- it ends the -> process.** ggml calls `abort()`, which no Java `catch` can intercept, and it happens even with -> `-ngl 0`: having the jar on the classpath is enough, because the backend registers and the device -> is used during setup whatever the layer count says. Measured on a Tesla T4 with driver -> 595.91.07, where that generation still shipped as PTX only. If you cannot control the driver, -> keep the jar off the classpath or pass `-Dnet.ladenthin.llama.backend=cpu`. +> **CUDA: the module ships native code for Turing through Blackwell, and a GPU outside that set +> needs a driver new enough for CUDA 13.4.** `sm_75` (T4, RTX 20xx), `sm_80` (A100), `sm_86`, +> `sm_89`, `sm_90` (H100), `sm_100` and Blackwell are compiled in, so those GPUs do not depend on +> the driver at all; anything newer falls back to the module's PTX, which the driver must be able to +> compile. **A driver too old for that does not fall back to the CPU -- it ends the process**: ggml +> calls `abort()`, which no Java `catch` can intercept, and it happens even with `-ngl 0`, because +> having the jar on the classpath is enough for the backend to register and the device to be used +> during setup whatever the layer count says. +> +> That was measured the hard way. On a Tesla T4 with driver 595.91.07 the previous module, which +> shipped Turing as PTX only, aborted the JVM in all three configurations; the same AMI, driver and +> GPU with native `sm_75` code runs cleanly and gives **20.6x prompt processing and 7.0x generation** +> over the CPU baseline. If you run a GPU generation newer than the list above and cannot control the +> driver, keep the jar off the classpath or pass `-Dnet.ladenthin.llama.backend=cpu`. > > **OpenVINO is opt-in, and that is why.** On a host *without* an Intel accelerator its CPU plugin -> opens perfectly well and contributes a device, so llama.cpp prefers it -- measured at -> **17x slower prompt processing and 172x slower generation** than the CPU modules (490.4/34.5 -> down to 29.4/**0.2** tokens per second). It therefore breaks the premise every other module jar -> keeps ("a module that cannot help is skipped") in both directions: it also fails to open at all -> without an OpenCL ICD loader, which its own runtime archive does not include. So its jar on the -> classpath does nothing until `net.ladenthin.llama.backend` names it, and the loader says so on -> stderr. +> opens perfectly well and contributes a device, so llama.cpp prefers it -- measured on an Intel +> Sapphire Rapids machine at **21.7x slower prompt processing and 336x slower generation** than the +> CPU modules (481.3/33.6 down to 22.2/**0.1** tokens per second, i.e. about ten seconds per token). +> It therefore breaks the premise every other module jar keeps ("a module that cannot help is +> skipped") in both directions: it also fails to open at all without an OpenCL ICD loader +> (`libOpenCL.so.1`, package `ocl-icd-libopencl1`), which its own runtime archive does not include. +> So its jar on the classpath does nothing until `net.ladenthin.llama.backend` names it, and the +> loader says so on stderr. > > **Vulkan is the one that behaves exactly as designed** and is the recommended portable choice: > without an ICD it is skipped, with throughput identical to the CPU baseline (201.8/24.1 against diff --git a/TODO.md b/TODO.md index b0fa3679..1774f192 100644 --- a/TODO.md +++ b/TODO.md @@ -210,12 +210,48 @@ files as each other, so they can go in one branch: exactly the failure the resolver exists for, and the one that stayed hidden in the core for hundreds of builds because a class-level assumption records no test entries. One test added. -**Round 2 -- needs round 0 to have produced artifacts.** -4. **A `g4dn` run proving a T4 works** with the new arch list (round 0 builds the module; the probe - package and harness already exist). -5. **The Windows extraction-reuse measurement** -- two JVMs, file locks, the one item on the - measurement list that is both cheap and still completely unmeasured. -6. **A slim-image load test**: build a container from `eclipse-temurin:21-jre-noble` and from +**Round 2 -- ~~needs round 0 to have produced artifacts~~ DONE 2026-10-12.** All three were measured +against the **published** snapshot jars (`20261012.035453-12`) rather than a build tree, so the +numbers are what a consumer resolving from Maven gets. Written up in +[`docs/benchmarks/gpu-module-jars.md`](docs/benchmarks/gpu-module-jars.md) and the new +[`docs/benchmarks/native-loading.md`](docs/benchmarks/native-loading.md); the account-specific record +is in the private workspace. Two AWS boots, **under 0.08 USD together**, both terminated and four +regions swept afterwards. +4. ~~**A `g4dn` run proving a T4 works** with the new arch list~~ -- **done, and the abort is gone.** + Same AMI, same driver 595.91.07, same Tesla T4; only the module changed. **Three exit-134 aborts + became three clean runs**, including `-ngl 0`, which was the most absurd of them since no layer was + on the GPU. The GPU gives **20.6x prompt processing and 7.0x generation** over the CPU baseline on + that machine. `cuobjdump` on the published module confirms native code for `sm_75`, `sm_80`, + `sm_86`, `sm_89`, `sm_90`, `sm_100`, `sm_120a` and `sm_121a` plus PTX for `sm_121a` -- so Turing, + A100 and Hopper no longer depend on the driver and anything newer than Blackwell still has a + fallback. The `a` suffixes are upstream's own rewrite of every `12x` architecture, not a defect in + the list this project passes. **The README's figures were corrected with it**: the CUDA paragraph + described the pre-fix state, and OpenVINO's cost was understated (see item 6). +5. ~~**The Windows extraction-reuse measurement**~~ -- **done, all four documented properties + confirmed.** Extracting 18 files / 39 MB costs **~810 ms per JVM start** (cold ~2175 ms against + warm ~1365 ms) and the reuse removes it -- per start, so also per surefire fork. The directory key + visibly carries the module set (`...-11b5000` against `...-11b5000-mf39ba978`). Two JVMs of one + build run concurrently without interfering. And the sharpest case: with an `RpcServer` holding the + library and the directory backdated past `STALE_EXTRACTION_AGE`, a sweeper left **exactly the five + files that JVM still had open** -- the library, the two pre-loaded ggml libraries, the one CPU + module ggml kept, and `ggml-rpc` -- while the 13 it had scored and closed went and were + re-extracted on the next start. The server kept running throughout. +6. ~~**The slim-image / distroless load test**~~ -- **done.** The published Linux jar declares only + what a JVM brings with it (`libc`, `libm`, `libdl`, `libpthread`, `libstdc++`, `libgcc_s`, run path + `$ORIGIN`, `GLIBC_2.27` at most) -- **`libgomp.so.1` is gone from the artifact**, not only from the + allowlist. `net.ladenthin.llama.Diagnose` from the published jars loads in + `gcr.io/distroless/java21-debian12` and `eclipse-temurin:21-jre`, and is refused by + `eclipse-temurin:21-jre-alpine` **by name**: `No native library found for os.name=Linux-Musl`. + That last row is not a defect -- musl was never a shipped platform -- and it was in the matrix on + purpose, because a measurement that only admits successes cannot show where the boundary is. + **Also measured, and it corrects this file and the README:** OpenVINO on an Intel Sapphire Rapids + instance costs **21.7x prompt processing and 336x generation** against the baseline of the same + machine (481.3/33.6 down to 22.2/**0.1**), not the 17x/172x recorded from the earlier machine. + Without an OpenCL ICD the module does not open at all (2 backends, full CPU speed); with + `ocl-icd-libopencl1` it registers a third and collapses. And with the jar on the classpath but + `net.ladenthin.llama.backend` unset the loader says `is on the classpath but not used` and the + machine runs at full speed -- **the only hardware proof the opt-in decision has**, since CI has no + Intel accelerator and nothing happening there proves nothing. distroless Java, drop the classes jar plus the Linux library jar in, and load. This is the check that CI *structurally cannot* do, because it runs on full runners -- and it is the direct verification of what round 0's dependency check only implies. diff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.md index f9addbbd..04874d54 100644 --- a/docs/benchmarks/README.md +++ b/docs/benchmarks/README.md @@ -15,6 +15,7 @@ themselves live in [`../../TODO.md`](../../TODO.md). |---|---| | [`cpu-backend-modules.md`](cpu-backend-modules.md) | What each of the 22 shipped CPU backend modules is worth, per microarchitecture — and whether `ggml_backend_load_best` picks the fastest one | | [`gpu-module-jars.md`](gpu-module-jars.md) | What happens when a GPU module jar is actually *used* -- two findings that contradict the "a module that cannot help is skipped" premise | +| [`native-loading.md`](native-loading.md) | What extracting the natives costs and what the reuse saves, how a running JVM's locked files behave, and which container base images can load the Linux jar | | [`ec2-harness.md`](ec2-harness.md) | How a measurement is run on rented hardware, and the invariants that came out of getting it wrong | ## Why these exist at all diff --git a/docs/benchmarks/ec2-harness.md b/docs/benchmarks/ec2-harness.md index 262038f1..dc9d2c8b 100644 --- a/docs/benchmarks/ec2-harness.md +++ b/docs/benchmarks/ec2-harness.md @@ -91,6 +91,14 @@ secret lands on a throwaway machine). The EC2 console log is **not** a usable ch language that treats `\f` and `\b` as escapes becomes a form feed or a backspace, and the resulting error (`Could not find or load main class C:\pixpkg\…`) costs a full boot. Build such paths from `chr(92)` or with forward slashes. +10. **The harness is the first suspect, not the artifact.** Two measurements this project ran looked + like product defects and were neither. A Windows classpath built in a POSIX shell had `C:/…` + rewritten to `C` plus `\Users\…`, so Java received broken entries and the loader correctly + reported no natives jar on the classpath (`MSYS_NO_PATHCONV=1` fixes it). And a local probe + declared all five SDK layouts broken because `chmod +x` does not stick on that filesystem, so + `[ -x ]` answered false for every fixture -- including the one that demonstrably works in CI. In + both cases the tool under test was telling the truth. Reproduce the *expected* case first; if even + that fails, the harness is wrong. ## Cost, measured @@ -114,5 +122,7 @@ not just the one that was used. A **GPU instance family starts at a zero vCPU quota** on a fresh account. Raising it is a support request measured in hours to days; the On-Demand and Spot quotas are **independent**, with separate -approvals, so filing both is worth it. Note that a Spot instance can only be terminated, never +approvals, so filing both is worth it. Both were granted here within a day, which is what finally +made the GPU findings in [`gpu-module-jars.md`](gpu-module-jars.md) measurable -- and the second GPU +run of this project cost under 0.05 USD, because the probe needed four minutes once it started. Note that a Spot instance can only be terminated, never stopped — which does not matter for a measurement that terminates anyway. diff --git a/docs/benchmarks/gpu-module-jars.md b/docs/benchmarks/gpu-module-jars.md index 6616b70c..706d842a 100644 --- a/docs/benchmarks/gpu-module-jars.md +++ b/docs/benchmarks/gpu-module-jars.md @@ -22,6 +22,45 @@ both are things a consumer will hit, and neither is currently documented. ## 1 · `cuda13` on a Tesla T4: it does not fall back, it aborts the JVM +> ### Resolved, and re-measured on the same hardware (2026-10-12) +> +> The module now ships **native code for `sm_75`** instead of PTX only, so the driver has nothing to +> JIT and the abort cannot occur. Verified by re-running the identical probe: **same AMI, same driver +> 595.91.07, same Tesla T4 — only the module changed.** +> +> | run | before (PTX only) | after (`sm_75` cubin) | +> |---|---|---| +> | automatic, `-ngl 99` | **exit 134** | **exit 0** · 29/29 layers · pp **4969.0** / tg **232.0** | +> | `cuda13` forced, `-ngl 99` | **exit 134** | **exit 0** · 29/29 · pp **4973.4** / tg **228.3** | +> | `cuda13` forced, `-ngl 0` | **exit 134** | **exit 0** · 0/29 · pp 1244.3 / tg 31.4 | +> | `vulkan` forced (control) | exit 0 · 201.8 / 24.1 | exit 0 · 240.3 / 32.6 | +> | `cpu` only (baseline) | exit 0 · 202.2 / 24.1 | exit 0 · 240.9 / 33.3 | +> +> Three aborts became three clean runs, and the T4 delivers **20.6x prompt processing** and **7.0x +> generation** over the CPU baseline on the same machine. That `-ngl 0` runs again closes the case: +> it was the most absurd of the three, since not one layer was on the GPU. +> +> **What the shipped module carries now**, read with `cuobjdump` from the published jar: +> +> | | architectures | +> |---|---| +> | native code (cubin) | `sm_75` `sm_80` `sm_86` `sm_89` `sm_90` `sm_100` `sm_120a` `sm_121a` — 144 each | +> | PTX (the JIT reserve) | `sm_121a`, 144 files | +> +> So Turing, A100 and Hopper no longer depend on the driver, and anything newer than Blackwell still +> has PTX to fall back on. The `a` suffixes are upstream's own doing rather than a defect in the list +> this project passes: `ggml/src/ggml-cuda/CMakeLists.txt` rewrites every `12x` architecture to +> `12xa`, deliberately, because `120a-real` works with any CMake version. +> +> **One method note, because it nearly produced a wrong conclusion:** a first scan reported *no* PTX, +> because it searched for `compute_`. `cuobjdump` names PTX files `libggml-cuda.N.sm_121a.ptx` — with +> `sm_`. Read the tool's raw output before concluding from a pattern. +> +> The original finding follows, unchanged, as the record of what the defect was. + +### The defect as it was found + + `g4dn.xlarge`, Tesla T4 (compute capability **7.5**, Turing), NVIDIA driver **595.91.07** (the AWS Deep Learning Base AMI), the `cuda13-linux-x86-64` module jar next to the modular Linux CPU library. @@ -137,8 +176,38 @@ install everything OpenVINO documents and still get nothing. `CLAUDE.md` names this dependency **for Windows** (`jllama-extras.txt` carrying OpenVINO's `OpenCL.dll`), where the OS supplies the loader anyway. For Linux it is undocumented. +### Re-measured 2026-10-12, and the README's figures were too kind + +On an `m7i.xlarge` (Xeon Platinum 8488C, Sapphire Rapids, 4 cores, no Intel accelerator), Ubuntu +24.04, the OpenVINO 2026.4.1 archive installed exactly as the CI job installs it, against the +**published** jars: + +| run | OpenCL ICD | backends registered | pp512 | tg128 | +|---|---|---:|---:|---:| +| `openvino` named, **no ICD** | absent | **2** | 497.2 | 36.4 | +| `openvino` named, **ICD installed** | present | **3** | **22.2** | **0.1** | +| jar on the classpath, property **unset** | present | **2** | 502.9 | 36.2 | +| `cpu` named (baseline) | present | 2 | 481.3 | 33.6 | + +Against the baseline on that machine the cost is **21.7x on prompt processing and 336x on +generation** — worse than the 17x/172x this page recorded from the earlier machine, so the README's +numbers understate it. Generation falls to **0.1 tokens per second**, roughly ten seconds per token, +with 27,639 log lines mentioning OpenVINO. + +Two things this settles that the earlier run could not: + +* **Without the ICD the module does not open at all.** The loader places it (`with GPU module(s) + openvino`) and ggml reports two backends, with throughput identical to the CPU baseline. The + OpenVINO runtime archive does not ship `libOpenCL.so.1`; `ocl-icd-libopencl1` is what changes the + answer, and installing it is what makes the regression appear. +* **The opt-in holds.** With the jar on the classpath and `net.ladenthin.llama.backend` unset, the + loader says `is on the classpath but not used`, two backends register and the machine runs at full + speed. This is the **first proof on hardware where OpenVINO could genuinely have taken over** — in + CI there is no Intel accelerator, so "nothing happened" proves nothing there. + ### What to do about it + Recorded in `TODO.md` as the owner's call, with a recommendation: **make the OpenVINO module opt-in** rather than additive. The premise of additive module jars is "a module that cannot help is skipped, one that can help is used" — OpenVINO satisfies neither on a CPU-only host. Failing that, the README diff --git a/docs/benchmarks/native-loading.md b/docs/benchmarks/native-loading.md new file mode 100644 index 00000000..0edfd7f7 --- /dev/null +++ b/docs/benchmarks/native-loading.md @@ -0,0 +1,116 @@ + + +# What loading the natives costs, and where it works at all + +Measured **2026-10-12** against the **published** 5.2.0-SNAPSHOT jars (`20261012.035453-12`, llama.cpp +b11558) rather than a build tree — so these numbers are what a consumer resolving from Maven actually +gets. + +Two questions that had never been measured: what the extraction directory's reuse is worth on +Windows, and which container base images can load the Linux jar at all. + +## The extraction directory on Windows + +`LlamaLoader` extracts the platform's library directory into a per-build subdirectory of the temp +directory and reuses it across JVM starts. Four properties are documented in `CLAUDE.md`'s +"Modular natives"; all four are confirmed below. + +**Method.** The published `cpu-windows-x86-64` jar plus the classes jar and its runtime dependencies, +`net.ladenthin.llama.Diagnose` as the entry point (it loads the library and exits), an isolated +`-Dnet.ladenthin.llama.tmpdir` so the starting state is controlled, three runs each. Windows 11, an +on-access scanner active. + +### The reuse is worth about 0.8 s per JVM start + +| | run 1 | run 2 | run 3 | +|---|---:|---:|---:| +| **cold** (directory cleared first) | 4854 ms | 2190 ms | 2159 ms | +| **warm** (directory kept) | 1376 ms | 1345 ms | 1376 ms | + +The first cold run is the file cache warming, not the measurement; the representative cold figure is +~2175 ms against ~1365 ms warm, so **extracting 18 files / 39 MB costs ~810 ms** and the reuse branch +removes it. That is per JVM start, and therefore also per surefire fork. + +### The key includes the module set, visibly + +| classpath | directory | +|---|---| +| library jar only | `jllama-backend-cpu-4fc8af71-11b5000` | +| library jar + `vulkan-windows-x86-64` | `jllama-backend-cpu-4fc8af71-11b5000-mf39ba978` | + +Two classpaths with different module sets therefore never share a directory, which is what stops one +JVM's set from being mixed into another's. + +### Two JVMs of the same build at once + +Both started concurrently from the same classpath, both exit 0, both report `load: ok`. + +### A running JVM's files survive another JVM's sweep -- exactly the five it still holds + +The sharpest test, because it is the one where a wrong answer corrupts a running process. An +`RpcServer` was left running (it loads the library and listens), its directory's mtime was backdated +by 25 minutes so that `STALE_EXTRACTION_AGE` (10 min) had passed, and a second JVM with a *different* +module set — so the first directory is "not on this classpath" — was started. + +**18 files before, 5 after.** The five survivors are precisely the ones the running JVM still had +open: + +| survivor | why it was locked | +|---|---| +| `jllama.dll` | the library itself | +| `ggml.dll`, `ggml-base.dll` | pre-loaded by full path via `jllama-extras.txt` | +| `ggml-cpu-haswell.dll` | **the CPU module ggml scored best and kept** | +| `ggml-rpc.dll` | in use by the RPC server | + +The 13 that went are exactly the CPU modules ggml opens to score and closes again — no longer locked, +so removable. **The `RpcServer` kept running throughout.** The next start of that build re-extracted +the 13 missing files (back to 18) in 1852 ms, between the warm and cold figures above, as copying 13 +of 18 files should be. + +So the documented behaviour holds in full: the cache is shared, a live JVM is never damaged by it, and +what a sweep takes is re-created on demand. + +## Which base images can load the Linux jar + +The published `cpu-linux-x86-64` jar declares, across all 18 libraries (read with the project's own +ELF parser, `buildcheck/nativedeps`): + +``` +ld-linux-x86-64.so.2 libc.so.6 libdl.so.2 libgcc_s.so.1 libm.so.6 +libpthread.so.0 libstdc++.so.6 + the siblings next to it +run path: $ORIGIN (every library) glibc: GLIBC_2.27 at most +``` + +**`libgomp.so.1` is gone** — the dependency that made the jar fail on a plain Ubuntu server image +(`UnsatisfiedLinkError: The native library of backend 'cpu' could not be loaded`, which reads like a +packaging defect and was a missing distro package). What remains is what a JVM brings with it. + +Verified by loading `net.ladenthin.llama.Diagnose` from the published jars inside three containers on +an EC2 instance: + +| image | result | +|---|---| +| `gcr.io/distroless/java21-debian12` | **loads** | +| `eclipse-temurin:21-jre` | **loads** | +| `eclipse-temurin:21-jre-alpine` | **refused**, and correctly: `No native library found for os.name=Linux-Musl, os.arch=x86_64` | + +The Alpine row is the one worth keeping. It is **not** a defect: `OSInfo` detects musl, looks for a +`Linux-Musl/x86_64` directory, finds none and says so by name. musl was never a shipped platform; what +is new is that the failure is now a named one rather than a dynamic-linker error. A measurement that +only admits successes cannot show where the boundary is, which is why the image that must fail was +part of the matrix. + +## What these runs cannot show + +* **One Windows machine**, whose CPU selects `ggml-cpu-haswell`. A machine that selects a different + module would lock a different file; the mechanism is the same, the survivor list is not. +* **The timings include JVM startup** (~270 ms measured with the natives absent), which is why the + cold-minus-warm difference is quoted rather than either figure alone. +* **No inference.** `Diagnose` loads the library and exits; throughput is measured elsewhere + ([`cpu-backend-modules.md`](cpu-backend-modules.md)). +* **The container rows prove loading, not running a model** — the same distinction the GPU page makes: + a successful load has repeatedly proven less than it looks.