From 4118b24ade2a26d4d1c62af99671deb38baa4f47 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Mon, 12 Oct 2026 09:44:43 +0200 Subject: [PATCH 1/2] Round 2 measured: the T4 abort is gone, and what loading the natives costs All three round-2 questions answered against the PUBLISHED 5.2.0-SNAPSHOT jars rather than a build tree, so the numbers are what a consumer resolving from Maven gets. The CUDA finding is resolved and re-measured on the same hardware: same AMI, same driver 595.91.07, same Tesla T4, only the module changed. Three exit-134 aborts became three clean runs, and the GPU delivers 20.6x prompt processing and 7.0x generation over the CPU baseline on that machine. That -ngl 0 runs again closes the case, since it was the most absurd of the three. cuobjdump on the published module shows native code for sm_75, sm_80, sm_86, sm_89, sm_90, sm_100, sm_120a and sm_121a plus PTX for sm_121a, so Turing, A100 and Hopper no longer depend on the driver and anything newer than Blackwell still has a fallback. The a suffixes are upstream's own rewrite of every 12x architecture, not a defect in the list this project passes. A method note went in with it, because it nearly produced a wrong conclusion: a first scan reported no PTX because it searched for compute_, while cuobjdump names those files sm_121a.ptx. OpenVINO was re-measured on an Intel Sapphire Rapids instance and the README's figures turn out too kind: 21.7x slower prompt processing and 336x slower generation against the baseline of the same machine, not 17x/172x. Two things the earlier run could not settle are settled now -- without an OpenCL ICD the module does not open at all, and with the jar on the classpath but the property unset the loader leaves it out and the machine runs at full speed. That last one is the only hardware proof the opt-in decision has, because CI has no Intel accelerator and nothing happening there proves nothing. New page, native-loading.md: extracting 18 files and 39 MB costs about 810 ms per JVM start on Windows and the reuse removes it; the directory key visibly includes the module set; two JVMs of one build run concurrently; and the sharpest case, a sweep against a backdated directory while an RpcServer holds it, leaves exactly the five files that JVM still had open -- the library, the two pre-loaded ggml libraries, the one CPU module ggml kept and ggml-rpc -- while the 13 it had scored and closed go and are re-extracted on the next start. The server kept running throughout. The page also records which container base images can load the Linux jar now that libgomp is gone, including the musl image that must fail and does so by name. The harness page gains a tenth invariant from two wrong conclusions this session: the harness is the first suspect, not the artifact. A Windows classpath mangled by the POSIX shell made the loader correctly report no natives jar, and a local probe called five SDK layouts broken because chmod +x does not stick on that filesystem. Reproduce the expected case first; if even that fails, the harness is wrong. No private detail reaches these pages: no bucket, account, instance id, AMI of the account's own making or presigned URL. The account-specific record is in the private workspace. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01E2h8gXyyE5UeimkL9vQv9G --- docs/benchmarks/README.md | 1 + docs/benchmarks/ec2-harness.md | 12 ++- docs/benchmarks/gpu-module-jars.md | 69 +++++++++++++++++ docs/benchmarks/native-loading.md | 116 +++++++++++++++++++++++++++++ 4 files changed, 197 insertions(+), 1 deletion(-) create mode 100644 docs/benchmarks/native-loading.md diff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.md index f9addbbd..04874d54 100644 --- a/docs/benchmarks/README.md +++ b/docs/benchmarks/README.md @@ -15,6 +15,7 @@ themselves live in [`../../TODO.md`](../../TODO.md). |---|---| | [`cpu-backend-modules.md`](cpu-backend-modules.md) | What each of the 22 shipped CPU backend modules is worth, per microarchitecture — and whether `ggml_backend_load_best` picks the fastest one | | [`gpu-module-jars.md`](gpu-module-jars.md) | What happens when a GPU module jar is actually *used* -- two findings that contradict the "a module that cannot help is skipped" premise | +| [`native-loading.md`](native-loading.md) | What extracting the natives costs and what the reuse saves, how a running JVM's locked files behave, and which container base images can load the Linux jar | | [`ec2-harness.md`](ec2-harness.md) | How a measurement is run on rented hardware, and the invariants that came out of getting it wrong | ## Why these exist at all diff --git a/docs/benchmarks/ec2-harness.md b/docs/benchmarks/ec2-harness.md index 262038f1..dc9d2c8b 100644 --- a/docs/benchmarks/ec2-harness.md +++ b/docs/benchmarks/ec2-harness.md @@ -91,6 +91,14 @@ secret lands on a throwaway machine). The EC2 console log is **not** a usable ch language that treats `\f` and `\b` as escapes becomes a form feed or a backspace, and the resulting error (`Could not find or load main class C:\pixpkg\…`) costs a full boot. Build such paths from `chr(92)` or with forward slashes. +10. **The harness is the first suspect, not the artifact.** Two measurements this project ran looked + like product defects and were neither. A Windows classpath built in a POSIX shell had `C:/…` + rewritten to `C` plus `\Users\…`, so Java received broken entries and the loader correctly + reported no natives jar on the classpath (`MSYS_NO_PATHCONV=1` fixes it). And a local probe + declared all five SDK layouts broken because `chmod +x` does not stick on that filesystem, so + `[ -x ]` answered false for every fixture -- including the one that demonstrably works in CI. In + both cases the tool under test was telling the truth. Reproduce the *expected* case first; if even + that fails, the harness is wrong. ## Cost, measured @@ -114,5 +122,7 @@ not just the one that was used. A **GPU instance family starts at a zero vCPU quota** on a fresh account. Raising it is a support request measured in hours to days; the On-Demand and Spot quotas are **independent**, with separate -approvals, so filing both is worth it. Note that a Spot instance can only be terminated, never +approvals, so filing both is worth it. Both were granted here within a day, which is what finally +made the GPU findings in [`gpu-module-jars.md`](gpu-module-jars.md) measurable -- and the second GPU +run of this project cost under 0.05 USD, because the probe needed four minutes once it started. Note that a Spot instance can only be terminated, never stopped — which does not matter for a measurement that terminates anyway. diff --git a/docs/benchmarks/gpu-module-jars.md b/docs/benchmarks/gpu-module-jars.md index 6616b70c..706d842a 100644 --- a/docs/benchmarks/gpu-module-jars.md +++ b/docs/benchmarks/gpu-module-jars.md @@ -22,6 +22,45 @@ both are things a consumer will hit, and neither is currently documented. ## 1 · `cuda13` on a Tesla T4: it does not fall back, it aborts the JVM +> ### Resolved, and re-measured on the same hardware (2026-10-12) +> +> The module now ships **native code for `sm_75`** instead of PTX only, so the driver has nothing to +> JIT and the abort cannot occur. Verified by re-running the identical probe: **same AMI, same driver +> 595.91.07, same Tesla T4 — only the module changed.** +> +> | run | before (PTX only) | after (`sm_75` cubin) | +> |---|---|---| +> | automatic, `-ngl 99` | **exit 134** | **exit 0** · 29/29 layers · pp **4969.0** / tg **232.0** | +> | `cuda13` forced, `-ngl 99` | **exit 134** | **exit 0** · 29/29 · pp **4973.4** / tg **228.3** | +> | `cuda13` forced, `-ngl 0` | **exit 134** | **exit 0** · 0/29 · pp 1244.3 / tg 31.4 | +> | `vulkan` forced (control) | exit 0 · 201.8 / 24.1 | exit 0 · 240.3 / 32.6 | +> | `cpu` only (baseline) | exit 0 · 202.2 / 24.1 | exit 0 · 240.9 / 33.3 | +> +> Three aborts became three clean runs, and the T4 delivers **20.6x prompt processing** and **7.0x +> generation** over the CPU baseline on the same machine. That `-ngl 0` runs again closes the case: +> it was the most absurd of the three, since not one layer was on the GPU. +> +> **What the shipped module carries now**, read with `cuobjdump` from the published jar: +> +> | | architectures | +> |---|---| +> | native code (cubin) | `sm_75` `sm_80` `sm_86` `sm_89` `sm_90` `sm_100` `sm_120a` `sm_121a` — 144 each | +> | PTX (the JIT reserve) | `sm_121a`, 144 files | +> +> So Turing, A100 and Hopper no longer depend on the driver, and anything newer than Blackwell still +> has PTX to fall back on. The `a` suffixes are upstream's own doing rather than a defect in the list +> this project passes: `ggml/src/ggml-cuda/CMakeLists.txt` rewrites every `12x` architecture to +> `12xa`, deliberately, because `120a-real` works with any CMake version. +> +> **One method note, because it nearly produced a wrong conclusion:** a first scan reported *no* PTX, +> because it searched for `compute_`. `cuobjdump` names PTX files `libggml-cuda.N.sm_121a.ptx` — with +> `sm_`. Read the tool's raw output before concluding from a pattern. +> +> The original finding follows, unchanged, as the record of what the defect was. + +### The defect as it was found + + `g4dn.xlarge`, Tesla T4 (compute capability **7.5**, Turing), NVIDIA driver **595.91.07** (the AWS Deep Learning Base AMI), the `cuda13-linux-x86-64` module jar next to the modular Linux CPU library. @@ -137,8 +176,38 @@ install everything OpenVINO documents and still get nothing. `CLAUDE.md` names this dependency **for Windows** (`jllama-extras.txt` carrying OpenVINO's `OpenCL.dll`), where the OS supplies the loader anyway. For Linux it is undocumented. +### Re-measured 2026-10-12, and the README's figures were too kind + +On an `m7i.xlarge` (Xeon Platinum 8488C, Sapphire Rapids, 4 cores, no Intel accelerator), Ubuntu +24.04, the OpenVINO 2026.4.1 archive installed exactly as the CI job installs it, against the +**published** jars: + +| run | OpenCL ICD | backends registered | pp512 | tg128 | +|---|---|---:|---:|---:| +| `openvino` named, **no ICD** | absent | **2** | 497.2 | 36.4 | +| `openvino` named, **ICD installed** | present | **3** | **22.2** | **0.1** | +| jar on the classpath, property **unset** | present | **2** | 502.9 | 36.2 | +| `cpu` named (baseline) | present | 2 | 481.3 | 33.6 | + +Against the baseline on that machine the cost is **21.7x on prompt processing and 336x on +generation** — worse than the 17x/172x this page recorded from the earlier machine, so the README's +numbers understate it. Generation falls to **0.1 tokens per second**, roughly ten seconds per token, +with 27,639 log lines mentioning OpenVINO. + +Two things this settles that the earlier run could not: + +* **Without the ICD the module does not open at all.** The loader places it (`with GPU module(s) + openvino`) and ggml reports two backends, with throughput identical to the CPU baseline. The + OpenVINO runtime archive does not ship `libOpenCL.so.1`; `ocl-icd-libopencl1` is what changes the + answer, and installing it is what makes the regression appear. +* **The opt-in holds.** With the jar on the classpath and `net.ladenthin.llama.backend` unset, the + loader says `is on the classpath but not used`, two backends register and the machine runs at full + speed. This is the **first proof on hardware where OpenVINO could genuinely have taken over** — in + CI there is no Intel accelerator, so "nothing happened" proves nothing there. + ### What to do about it + Recorded in `TODO.md` as the owner's call, with a recommendation: **make the OpenVINO module opt-in** rather than additive. The premise of additive module jars is "a module that cannot help is skipped, one that can help is used" — OpenVINO satisfies neither on a CPU-only host. Failing that, the README diff --git a/docs/benchmarks/native-loading.md b/docs/benchmarks/native-loading.md new file mode 100644 index 00000000..0edfd7f7 --- /dev/null +++ b/docs/benchmarks/native-loading.md @@ -0,0 +1,116 @@ + + +# What loading the natives costs, and where it works at all + +Measured **2026-10-12** against the **published** 5.2.0-SNAPSHOT jars (`20261012.035453-12`, llama.cpp +b11558) rather than a build tree — so these numbers are what a consumer resolving from Maven actually +gets. + +Two questions that had never been measured: what the extraction directory's reuse is worth on +Windows, and which container base images can load the Linux jar at all. + +## The extraction directory on Windows + +`LlamaLoader` extracts the platform's library directory into a per-build subdirectory of the temp +directory and reuses it across JVM starts. Four properties are documented in `CLAUDE.md`'s +"Modular natives"; all four are confirmed below. + +**Method.** The published `cpu-windows-x86-64` jar plus the classes jar and its runtime dependencies, +`net.ladenthin.llama.Diagnose` as the entry point (it loads the library and exits), an isolated +`-Dnet.ladenthin.llama.tmpdir` so the starting state is controlled, three runs each. Windows 11, an +on-access scanner active. + +### The reuse is worth about 0.8 s per JVM start + +| | run 1 | run 2 | run 3 | +|---|---:|---:|---:| +| **cold** (directory cleared first) | 4854 ms | 2190 ms | 2159 ms | +| **warm** (directory kept) | 1376 ms | 1345 ms | 1376 ms | + +The first cold run is the file cache warming, not the measurement; the representative cold figure is +~2175 ms against ~1365 ms warm, so **extracting 18 files / 39 MB costs ~810 ms** and the reuse branch +removes it. That is per JVM start, and therefore also per surefire fork. + +### The key includes the module set, visibly + +| classpath | directory | +|---|---| +| library jar only | `jllama-backend-cpu-4fc8af71-11b5000` | +| library jar + `vulkan-windows-x86-64` | `jllama-backend-cpu-4fc8af71-11b5000-mf39ba978` | + +Two classpaths with different module sets therefore never share a directory, which is what stops one +JVM's set from being mixed into another's. + +### Two JVMs of the same build at once + +Both started concurrently from the same classpath, both exit 0, both report `load: ok`. + +### A running JVM's files survive another JVM's sweep -- exactly the five it still holds + +The sharpest test, because it is the one where a wrong answer corrupts a running process. An +`RpcServer` was left running (it loads the library and listens), its directory's mtime was backdated +by 25 minutes so that `STALE_EXTRACTION_AGE` (10 min) had passed, and a second JVM with a *different* +module set — so the first directory is "not on this classpath" — was started. + +**18 files before, 5 after.** The five survivors are precisely the ones the running JVM still had +open: + +| survivor | why it was locked | +|---|---| +| `jllama.dll` | the library itself | +| `ggml.dll`, `ggml-base.dll` | pre-loaded by full path via `jllama-extras.txt` | +| `ggml-cpu-haswell.dll` | **the CPU module ggml scored best and kept** | +| `ggml-rpc.dll` | in use by the RPC server | + +The 13 that went are exactly the CPU modules ggml opens to score and closes again — no longer locked, +so removable. **The `RpcServer` kept running throughout.** The next start of that build re-extracted +the 13 missing files (back to 18) in 1852 ms, between the warm and cold figures above, as copying 13 +of 18 files should be. + +So the documented behaviour holds in full: the cache is shared, a live JVM is never damaged by it, and +what a sweep takes is re-created on demand. + +## Which base images can load the Linux jar + +The published `cpu-linux-x86-64` jar declares, across all 18 libraries (read with the project's own +ELF parser, `buildcheck/nativedeps`): + +``` +ld-linux-x86-64.so.2 libc.so.6 libdl.so.2 libgcc_s.so.1 libm.so.6 +libpthread.so.0 libstdc++.so.6 + the siblings next to it +run path: $ORIGIN (every library) glibc: GLIBC_2.27 at most +``` + +**`libgomp.so.1` is gone** — the dependency that made the jar fail on a plain Ubuntu server image +(`UnsatisfiedLinkError: The native library of backend 'cpu' could not be loaded`, which reads like a +packaging defect and was a missing distro package). What remains is what a JVM brings with it. + +Verified by loading `net.ladenthin.llama.Diagnose` from the published jars inside three containers on +an EC2 instance: + +| image | result | +|---|---| +| `gcr.io/distroless/java21-debian12` | **loads** | +| `eclipse-temurin:21-jre` | **loads** | +| `eclipse-temurin:21-jre-alpine` | **refused**, and correctly: `No native library found for os.name=Linux-Musl, os.arch=x86_64` | + +The Alpine row is the one worth keeping. It is **not** a defect: `OSInfo` detects musl, looks for a +`Linux-Musl/x86_64` directory, finds none and says so by name. musl was never a shipped platform; what +is new is that the failure is now a named one rather than a dynamic-linker error. A measurement that +only admits successes cannot show where the boundary is, which is why the image that must fail was +part of the matrix. + +## What these runs cannot show + +* **One Windows machine**, whose CPU selects `ggml-cpu-haswell`. A machine that selects a different + module would lock a different file; the mechanism is the same, the survivor list is not. +* **The timings include JVM startup** (~270 ms measured with the natives absent), which is why the + cold-minus-warm difference is quoted rather than either figure alone. +* **No inference.** `Diagnose` loads the library and exits; throughput is measured elsewhere + ([`cpu-backend-modules.md`](cpu-backend-modules.md)). +* **The container rows prove loading, not running a model** — the same distinction the GPU page makes: + a successful load has repeatedly proven less than it looks. From 59cb0cb1fcc6d2783f19b478343a9a6c2c792bcc Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Mon, 12 Oct 2026 09:49:05 +0200 Subject: [PATCH 2/2] Correct the figures the measurements moved, and close round 2 Two numbers in the README were wrong, and one paragraph described a state that no longer exists. The CUDA paragraph was written before the T4 was re-measured: it told a reader that the driver must be new enough for CUDA 13.4, full stop. That is now only true for a GPU newer than the shipped set. The module compiles in sm_75 (T4, RTX 20xx), sm_80 (A100), sm_86, sm_89, sm_90 (H100), sm_100 and Blackwell, so those do not depend on the driver at all; only something newer falls back to PTX, and only there can a too-old driver still end the process. The paragraph now says that, and names what the measurement cost to learn: the previous module aborted the JVM in all three configurations on a Tesla T4 with driver 595.91.07, while the same AMI, driver and GPU with native sm_75 code runs cleanly at 20.6x prompt processing and 7.0x generation over the CPU baseline. OpenVINO's cost was understated. Re-measured on an Intel Sapphire Rapids instance it is 21.7x slower prompt processing and 336x slower generation against the baseline of the same machine, not 17x/172x -- 481.3/33.6 down to 22.2/0.1 tokens per second, about ten seconds per token. The ICD package name is in the text now too, since that is what a consumer has to install or deliberately not install. TODO.md closes round 2 with what each item answered rather than deleting it: the T4 abort and the architecture listing, the Windows extraction measurement including the five files a running JVM keeps locked, and the container matrix with the musl row that must fail and does so by name. Both write-ups are linked; no account detail is in any of it. check-natives.py 0 disagreements, check-patches.py 0 problems, 122 buildcheck tests green, and a search for bucket names, account ids, instance ids, AMIs and presigned URLs over README.md, TODO.md and docs/benchmarks finds nothing. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01E2h8gXyyE5UeimkL9vQv9G --- README.md | 38 ++++++++++++++++++++++---------------- TODO.md | 48 ++++++++++++++++++++++++++++++++++++++++++------ 2 files changed, 64 insertions(+), 22 deletions(-) diff --git a/README.md b/README.md index 6b9b7ccc..da20ee41 100644 --- a/README.md +++ b/README.md @@ -287,24 +287,30 @@ platforms you target, e.g. `cpu-linux-x86-64`. > **Two GPU jars need more than their runtime, and both were found by measurement rather than by > reading the code** ([`docs/benchmarks/gpu-module-jars.md`](docs/benchmarks/gpu-module-jars.md)). > -> **CUDA: the driver must be new enough for the toolkit the module was built with (13.4).** The -> module ships native code for the GPU generations listed in `CMAKE_CUDA_ARCHITECTURES` -> (Turing through Blackwell); a GPU outside that set runs only if the driver can compile the -> module's PTX, and **a driver too old to do so does not fall back to the CPU -- it ends the -> process.** ggml calls `abort()`, which no Java `catch` can intercept, and it happens even with -> `-ngl 0`: having the jar on the classpath is enough, because the backend registers and the device -> is used during setup whatever the layer count says. Measured on a Tesla T4 with driver -> 595.91.07, where that generation still shipped as PTX only. If you cannot control the driver, -> keep the jar off the classpath or pass `-Dnet.ladenthin.llama.backend=cpu`. +> **CUDA: the module ships native code for Turing through Blackwell, and a GPU outside that set +> needs a driver new enough for CUDA 13.4.** `sm_75` (T4, RTX 20xx), `sm_80` (A100), `sm_86`, +> `sm_89`, `sm_90` (H100), `sm_100` and Blackwell are compiled in, so those GPUs do not depend on +> the driver at all; anything newer falls back to the module's PTX, which the driver must be able to +> compile. **A driver too old for that does not fall back to the CPU -- it ends the process**: ggml +> calls `abort()`, which no Java `catch` can intercept, and it happens even with `-ngl 0`, because +> having the jar on the classpath is enough for the backend to register and the device to be used +> during setup whatever the layer count says. +> +> That was measured the hard way. On a Tesla T4 with driver 595.91.07 the previous module, which +> shipped Turing as PTX only, aborted the JVM in all three configurations; the same AMI, driver and +> GPU with native `sm_75` code runs cleanly and gives **20.6x prompt processing and 7.0x generation** +> over the CPU baseline. If you run a GPU generation newer than the list above and cannot control the +> driver, keep the jar off the classpath or pass `-Dnet.ladenthin.llama.backend=cpu`. > > **OpenVINO is opt-in, and that is why.** On a host *without* an Intel accelerator its CPU plugin -> opens perfectly well and contributes a device, so llama.cpp prefers it -- measured at -> **17x slower prompt processing and 172x slower generation** than the CPU modules (490.4/34.5 -> down to 29.4/**0.2** tokens per second). It therefore breaks the premise every other module jar -> keeps ("a module that cannot help is skipped") in both directions: it also fails to open at all -> without an OpenCL ICD loader, which its own runtime archive does not include. So its jar on the -> classpath does nothing until `net.ladenthin.llama.backend` names it, and the loader says so on -> stderr. +> opens perfectly well and contributes a device, so llama.cpp prefers it -- measured on an Intel +> Sapphire Rapids machine at **21.7x slower prompt processing and 336x slower generation** than the +> CPU modules (481.3/33.6 down to 22.2/**0.1** tokens per second, i.e. about ten seconds per token). +> It therefore breaks the premise every other module jar keeps ("a module that cannot help is +> skipped") in both directions: it also fails to open at all without an OpenCL ICD loader +> (`libOpenCL.so.1`, package `ocl-icd-libopencl1`), which its own runtime archive does not include. +> So its jar on the classpath does nothing until `net.ladenthin.llama.backend` names it, and the +> loader says so on stderr. > > **Vulkan is the one that behaves exactly as designed** and is the recommended portable choice: > without an ICD it is skipped, with throughput identical to the CPU baseline (201.8/24.1 against diff --git a/TODO.md b/TODO.md index b0fa3679..1774f192 100644 --- a/TODO.md +++ b/TODO.md @@ -210,12 +210,48 @@ files as each other, so they can go in one branch: exactly the failure the resolver exists for, and the one that stayed hidden in the core for hundreds of builds because a class-level assumption records no test entries. One test added. -**Round 2 -- needs round 0 to have produced artifacts.** -4. **A `g4dn` run proving a T4 works** with the new arch list (round 0 builds the module; the probe - package and harness already exist). -5. **The Windows extraction-reuse measurement** -- two JVMs, file locks, the one item on the - measurement list that is both cheap and still completely unmeasured. -6. **A slim-image load test**: build a container from `eclipse-temurin:21-jre-noble` and from +**Round 2 -- ~~needs round 0 to have produced artifacts~~ DONE 2026-10-12.** All three were measured +against the **published** snapshot jars (`20261012.035453-12`) rather than a build tree, so the +numbers are what a consumer resolving from Maven gets. Written up in +[`docs/benchmarks/gpu-module-jars.md`](docs/benchmarks/gpu-module-jars.md) and the new +[`docs/benchmarks/native-loading.md`](docs/benchmarks/native-loading.md); the account-specific record +is in the private workspace. Two AWS boots, **under 0.08 USD together**, both terminated and four +regions swept afterwards. +4. ~~**A `g4dn` run proving a T4 works** with the new arch list~~ -- **done, and the abort is gone.** + Same AMI, same driver 595.91.07, same Tesla T4; only the module changed. **Three exit-134 aborts + became three clean runs**, including `-ngl 0`, which was the most absurd of them since no layer was + on the GPU. The GPU gives **20.6x prompt processing and 7.0x generation** over the CPU baseline on + that machine. `cuobjdump` on the published module confirms native code for `sm_75`, `sm_80`, + `sm_86`, `sm_89`, `sm_90`, `sm_100`, `sm_120a` and `sm_121a` plus PTX for `sm_121a` -- so Turing, + A100 and Hopper no longer depend on the driver and anything newer than Blackwell still has a + fallback. The `a` suffixes are upstream's own rewrite of every `12x` architecture, not a defect in + the list this project passes. **The README's figures were corrected with it**: the CUDA paragraph + described the pre-fix state, and OpenVINO's cost was understated (see item 6). +5. ~~**The Windows extraction-reuse measurement**~~ -- **done, all four documented properties + confirmed.** Extracting 18 files / 39 MB costs **~810 ms per JVM start** (cold ~2175 ms against + warm ~1365 ms) and the reuse removes it -- per start, so also per surefire fork. The directory key + visibly carries the module set (`...-11b5000` against `...-11b5000-mf39ba978`). Two JVMs of one + build run concurrently without interfering. And the sharpest case: with an `RpcServer` holding the + library and the directory backdated past `STALE_EXTRACTION_AGE`, a sweeper left **exactly the five + files that JVM still had open** -- the library, the two pre-loaded ggml libraries, the one CPU + module ggml kept, and `ggml-rpc` -- while the 13 it had scored and closed went and were + re-extracted on the next start. The server kept running throughout. +6. ~~**The slim-image / distroless load test**~~ -- **done.** The published Linux jar declares only + what a JVM brings with it (`libc`, `libm`, `libdl`, `libpthread`, `libstdc++`, `libgcc_s`, run path + `$ORIGIN`, `GLIBC_2.27` at most) -- **`libgomp.so.1` is gone from the artifact**, not only from the + allowlist. `net.ladenthin.llama.Diagnose` from the published jars loads in + `gcr.io/distroless/java21-debian12` and `eclipse-temurin:21-jre`, and is refused by + `eclipse-temurin:21-jre-alpine` **by name**: `No native library found for os.name=Linux-Musl`. + That last row is not a defect -- musl was never a shipped platform -- and it was in the matrix on + purpose, because a measurement that only admits successes cannot show where the boundary is. + **Also measured, and it corrects this file and the README:** OpenVINO on an Intel Sapphire Rapids + instance costs **21.7x prompt processing and 336x generation** against the baseline of the same + machine (481.3/33.6 down to 22.2/**0.1**), not the 17x/172x recorded from the earlier machine. + Without an OpenCL ICD the module does not open at all (2 backends, full CPU speed); with + `ocl-icd-libopencl1` it registers a third and collapses. And with the jar on the classpath but + `net.ladenthin.llama.backend` unset the loader says `is on the classpath but not used` and the + machine runs at full speed -- **the only hardware proof the opt-in decision has**, since CI has no + Intel accelerator and nothing happening there proves nothing. distroless Java, drop the classes jar plus the Linux library jar in, and load. This is the check that CI *structurally cannot* do, because it runs on full runners -- and it is the direct verification of what round 0's dependency check only implies.