Round 2 measured: the T4 abort is gone, and what loading the natives costs

README.md

@@ -287,24 +287,30 @@ platforms you target, e.g. `cpu-linux-x86-64`.
 > **Two GPU jars need more than their runtime, and both were found by measurement rather than by
 > reading the code** ([`docs/benchmarks/gpu-module-jars.md`](docs/benchmarks/gpu-module-jars.md)).
 >
-> **CUDA: the driver must be new enough for the toolkit the module was built with (13.4).** The
-> module ships native code for the GPU generations listed in `CMAKE_CUDA_ARCHITECTURES`
-> (Turing through Blackwell); a GPU outside that set runs only if the driver can compile the
-> module's PTX, and **a driver too old to do so does not fall back to the CPU -- it ends the
-> process.** ggml calls `abort()`, which no Java `catch` can intercept, and it happens even with
-> `-ngl 0`: having the jar on the classpath is enough, because the backend registers and the device
-> is used during setup whatever the layer count says. Measured on a Tesla T4 with driver
-> 595.91.07, where that generation still shipped as PTX only. If you cannot control the driver,
-> keep the jar off the classpath or pass `-Dnet.ladenthin.llama.backend=cpu`.
+> **CUDA: the module ships native code for Turing through Blackwell, and a GPU outside that set
+> needs a driver new enough for CUDA 13.4.** `sm_75` (T4, RTX 20xx), `sm_80` (A100), `sm_86`,
+> `sm_89`, `sm_90` (H100), `sm_100` and Blackwell are compiled in, so those GPUs do not depend on
+> the driver at all; anything newer falls back to the module's PTX, which the driver must be able to
+> compile. **A driver too old for that does not fall back to the CPU -- it ends the process**: ggml
+> calls `abort()`, which no Java `catch` can intercept, and it happens even with `-ngl 0`, because
+> having the jar on the classpath is enough for the backend to register and the device to be used
+> during setup whatever the layer count says.
+>
+> That was measured the hard way. On a Tesla T4 with driver 595.91.07 the previous module, which
+> shipped Turing as PTX only, aborted the JVM in all three configurations; the same AMI, driver and
+> GPU with native `sm_75` code runs cleanly and gives **20.6x prompt processing and 7.0x generation**
+> over the CPU baseline. If you run a GPU generation newer than the list above and cannot control the
+> driver, keep the jar off the classpath or pass `-Dnet.ladenthin.llama.backend=cpu`.
 >
 > **OpenVINO is opt-in, and that is why.** On a host *without* an Intel accelerator its CPU plugin
-> opens perfectly well and contributes a device, so llama.cpp prefers it -- measured at
-> **17x slower prompt processing and 172x slower generation** than the CPU modules (490.4/34.5
-> down to 29.4/**0.2** tokens per second). It therefore breaks the premise every other module jar
-> keeps ("a module that cannot help is skipped") in both directions: it also fails to open at all
-> without an OpenCL ICD loader, which its own runtime archive does not include. So its jar on the
-> classpath does nothing until `net.ladenthin.llama.backend` names it, and the loader says so on
-> stderr.
+> opens perfectly well and contributes a device, so llama.cpp prefers it -- measured on an Intel
+> Sapphire Rapids machine at **21.7x slower prompt processing and 336x slower generation** than the
+> CPU modules (481.3/33.6 down to 22.2/**0.1** tokens per second, i.e. about ten seconds per token).
+> It therefore breaks the premise every other module jar keeps ("a module that cannot help is
+> skipped") in both directions: it also fails to open at all without an OpenCL ICD loader
+> (`libOpenCL.so.1`, package `ocl-icd-libopencl1`), which its own runtime archive does not include.
+> So its jar on the classpath does nothing until `net.ladenthin.llama.backend` names it, and the
+> loader says so on stderr.
 >
 > **Vulkan is the one that behaves exactly as designed** and is the recommended portable choice:
 > without an ICD it is skipped, with throughput identical to the CPU baseline (201.8/24.1 against

TODO.md

@@ -210,12 +210,48 @@ files as each other, so they can go in one branch:
    exactly the failure the resolver exists for, and the one that stayed hidden in the core for
    hundreds of builds because a class-level assumption records no test entries. One test added.
 
-**Round 2 -- needs round 0 to have produced artifacts.**
-4. **A `g4dn` run proving a T4 works** with the new arch list (round 0 builds the module; the probe
-   package and harness already exist).
-5. **The Windows extraction-reuse measurement** -- two JVMs, file locks, the one item on the
-   measurement list that is both cheap and still completely unmeasured.
-6. **A slim-image load test**: build a container from `eclipse-temurin:21-jre-noble` and from
+**Round 2 -- ~~needs round 0 to have produced artifacts~~ DONE 2026-10-12.** All three were measured
+against the **published** snapshot jars (`20261012.035453-12`) rather than a build tree, so the
+numbers are what a consumer resolving from Maven gets. Written up in
+[`docs/benchmarks/gpu-module-jars.md`](docs/benchmarks/gpu-module-jars.md) and the new
+[`docs/benchmarks/native-loading.md`](docs/benchmarks/native-loading.md); the account-specific record
+is in the private workspace. Two AWS boots, **under 0.08 USD together**, both terminated and four
+regions swept afterwards.
+4. ~~**A `g4dn` run proving a T4 works** with the new arch list~~ -- **done, and the abort is gone.**
+   Same AMI, same driver 595.91.07, same Tesla T4; only the module changed. **Three exit-134 aborts
+   became three clean runs**, including `-ngl 0`, which was the most absurd of them since no layer was
+   on the GPU. The GPU gives **20.6x prompt processing and 7.0x generation** over the CPU baseline on
+   that machine. `cuobjdump` on the published module confirms native code for `sm_75`, `sm_80`,
+   `sm_86`, `sm_89`, `sm_90`, `sm_100`, `sm_120a` and `sm_121a` plus PTX for `sm_121a` -- so Turing,
+   A100 and Hopper no longer depend on the driver and anything newer than Blackwell still has a
+   fallback. The `a` suffixes are upstream's own rewrite of every `12x` architecture, not a defect in
+   the list this project passes. **The README's figures were corrected with it**: the CUDA paragraph
+   described the pre-fix state, and OpenVINO's cost was understated (see item 6).
+5. ~~**The Windows extraction-reuse measurement**~~ -- **done, all four documented properties
+   confirmed.** Extracting 18 files / 39 MB costs **~810 ms per JVM start** (cold ~2175 ms against
+   warm ~1365 ms) and the reuse removes it -- per start, so also per surefire fork. The directory key
+   visibly carries the module set (`...-11b5000` against `...-11b5000-mf39ba978`). Two JVMs of one
+   build run concurrently without interfering. And the sharpest case: with an `RpcServer` holding the
+   library and the directory backdated past `STALE_EXTRACTION_AGE`, a sweeper left **exactly the five
+   files that JVM still had open** -- the library, the two pre-loaded ggml libraries, the one CPU
+   module ggml kept, and `ggml-rpc` -- while the 13 it had scored and closed went and were
+   re-extracted on the next start. The server kept running throughout.
+6. ~~**The slim-image / distroless load test**~~ -- **done.** The published Linux jar declares only
+   what a JVM brings with it (`libc`, `libm`, `libdl`, `libpthread`, `libstdc++`, `libgcc_s`, run path
+   `$ORIGIN`, `GLIBC_2.27` at most) -- **`libgomp.so.1` is gone from the artifact**, not only from the
+   allowlist. `net.ladenthin.llama.Diagnose` from the published jars loads in
+   `gcr.io/distroless/java21-debian12` and `eclipse-temurin:21-jre`, and is refused by
+   `eclipse-temurin:21-jre-alpine` **by name**: `No native library found for os.name=Linux-Musl`.
+   That last row is not a defect -- musl was never a shipped platform -- and it was in the matrix on
+   purpose, because a measurement that only admits successes cannot show where the boundary is.
+   **Also measured, and it corrects this file and the README:** OpenVINO on an Intel Sapphire Rapids
+   instance costs **21.7x prompt processing and 336x generation** against the baseline of the same
+   machine (481.3/33.6 down to 22.2/**0.1**), not the 17x/172x recorded from the earlier machine.
+   Without an OpenCL ICD the module does not open at all (2 backends, full CPU speed); with
+   `ocl-icd-libopencl1` it registers a third and collapses. And with the jar on the classpath but
+   `net.ladenthin.llama.backend` unset the loader says `is on the classpath but not used` and the
+   machine runs at full speed -- **the only hardware proof the opt-in decision has**, since CI has no
+   Intel accelerator and nothing happening there proves nothing.
    distroless Java, drop the classes jar plus the Linux library jar in, and load. This is the check
    that CI *structurally cannot* do, because it runs on full runners -- and it is the direct
    verification of what round 0's dependency check only implies.

docs/benchmarks/README.md

@@ -15,6 +15,7 @@ themselves live in [`../../TODO.md`](../../TODO.md).
 |---|---|
 | [`cpu-backend-modules.md`](cpu-backend-modules.md) | What each of the 22 shipped CPU backend modules is worth, per microarchitecture — and whether `ggml_backend_load_best` picks the fastest one |
 | [`gpu-module-jars.md`](gpu-module-jars.md) | What happens when a GPU module jar is actually *used* -- two findings that contradict the "a module that cannot help is skipped" premise |
+| [`native-loading.md`](native-loading.md) | What extracting the natives costs and what the reuse saves, how a running JVM's locked files behave, and which container base images can load the Linux jar |
 | [`ec2-harness.md`](ec2-harness.md) | How a measurement is run on rented hardware, and the invariants that came out of getting it wrong |
 
 ## Why these exist at all

docs/benchmarks/ec2-harness.md

@@ -91,6 +91,14 @@ secret lands on a throwaway machine). The EC2 console log is **not** a usable ch
    language that treats `\f` and `\b` as escapes becomes a form feed or a backspace, and the
    resulting error (`Could not find or load main class C:\pixpkg\…`) costs a full boot. Build such
    paths from `chr(92)` or with forward slashes.
+10. **The harness is the first suspect, not the artifact.** Two measurements this project ran looked
+   like product defects and were neither. A Windows classpath built in a POSIX shell had `C:/…`
+   rewritten to `C` plus `\Users\…`, so Java received broken entries and the loader correctly
+   reported no natives jar on the classpath (`MSYS_NO_PATHCONV=1` fixes it). And a local probe
+   declared all five SDK layouts broken because `chmod +x` does not stick on that filesystem, so
+   `[ -x ]` answered false for every fixture -- including the one that demonstrably works in CI. In
+   both cases the tool under test was telling the truth. Reproduce the *expected* case first; if even
+   that fails, the harness is wrong.
 
 ## Cost, measured
 
@@ -114,5 +122,7 @@ not just the one that was used.
 
 A **GPU instance family starts at a zero vCPU quota** on a fresh account. Raising it is a support
 request measured in hours to days; the On-Demand and Spot quotas are **independent**, with separate
-approvals, so filing both is worth it. Note that a Spot instance can only be terminated, never
+approvals, so filing both is worth it. Both were granted here within a day, which is what finally
+made the GPU findings in [`gpu-module-jars.md`](gpu-module-jars.md) measurable -- and the second GPU
+run of this project cost under 0.05 USD, because the probe needed four minutes once it started. Note that a Spot instance can only be terminated, never
 stopped — which does not matter for a measurement that terminates anyway.

docs/benchmarks/gpu-module-jars.md

@@ -22,6 +22,45 @@ both are things a consumer will hit, and neither is currently documented.
 
 ## 1 · `cuda13` on a Tesla T4: it does not fall back, it aborts the JVM
 
+> ### Resolved, and re-measured on the same hardware (2026-10-12)
+>
+> The module now ships **native code for `sm_75`** instead of PTX only, so the driver has nothing to
+> JIT and the abort cannot occur. Verified by re-running the identical probe: **same AMI, same driver
+> 595.91.07, same Tesla T4 — only the module changed.**
+>
+> | run | before (PTX only) | after (`sm_75` cubin) |
+> |---|---|---|
+> | automatic, `-ngl 99` | **exit 134** | **exit 0** · 29/29 layers · pp **4969.0** / tg **232.0** |
+> | `cuda13` forced, `-ngl 99` | **exit 134** | **exit 0** · 29/29 · pp **4973.4** / tg **228.3** |
+> | `cuda13` forced, `-ngl 0` | **exit 134** | **exit 0** · 0/29 · pp 1244.3 / tg 31.4 |
+> | `vulkan` forced (control) | exit 0 · 201.8 / 24.1 | exit 0 · 240.3 / 32.6 |
+> | `cpu` only (baseline) | exit 0 · 202.2 / 24.1 | exit 0 · 240.9 / 33.3 |
+>
+> Three aborts became three clean runs, and the T4 delivers **20.6x prompt processing** and **7.0x
+> generation** over the CPU baseline on the same machine. That `-ngl 0` runs again closes the case:
+> it was the most absurd of the three, since not one layer was on the GPU.
+>
+> **What the shipped module carries now**, read with `cuobjdump` from the published jar:
+>
+> | | architectures |
+> |---|---|
+> | native code (cubin) | `sm_75` `sm_80` `sm_86` `sm_89` `sm_90` `sm_100` `sm_120a` `sm_121a` — 144 each |
+> | PTX (the JIT reserve) | `sm_121a`, 144 files |
+>
+> So Turing, A100 and Hopper no longer depend on the driver, and anything newer than Blackwell still
+> has PTX to fall back on. The `a` suffixes are upstream's own doing rather than a defect in the list
+> this project passes: `ggml/src/ggml-cuda/CMakeLists.txt` rewrites every `12x` architecture to
+> `12xa`, deliberately, because `120a-real` works with any CMake version.
+>
+> **One method note, because it nearly produced a wrong conclusion:** a first scan reported *no* PTX,
+> because it searched for `compute_`. `cuobjdump` names PTX files `libggml-cuda.N.sm_121a.ptx` — with
+> `sm_`. Read the tool's raw output before concluding from a pattern.
+>
+> The original finding follows, unchanged, as the record of what the defect was.
+
+### The defect as it was found
+
+
 `g4dn.xlarge`, Tesla T4 (compute capability **7.5**, Turing), NVIDIA driver **595.91.07** (the AWS
 Deep Learning Base AMI), the `cuda13-linux-x86-64` module jar next to the modular Linux CPU library.
 
@@ -137,8 +176,38 @@ install everything OpenVINO documents and still get nothing.
 `CLAUDE.md` names this dependency **for Windows** (`jllama-extras.txt` carrying OpenVINO's
 `OpenCL.dll`), where the OS supplies the loader anyway. For Linux it is undocumented.
 
+### Re-measured 2026-10-12, and the README's figures were too kind
+
+On an `m7i.xlarge` (Xeon Platinum 8488C, Sapphire Rapids, 4 cores, no Intel accelerator), Ubuntu
+24.04, the OpenVINO 2026.4.1 archive installed exactly as the CI job installs it, against the
+**published** jars:
+
+| run | OpenCL ICD | backends registered | pp512 | tg128 |
+|---|---|---:|---:|---:|
+| `openvino` named, **no ICD** | absent | **2** | 497.2 | 36.4 |
+| `openvino` named, **ICD installed** | present | **3** | **22.2** | **0.1** |
+| jar on the classpath, property **unset** | present | **2** | 502.9 | 36.2 |
+| `cpu` named (baseline) | present | 2 | 481.3 | 33.6 |
+
+Against the baseline on that machine the cost is **21.7x on prompt processing and 336x on
+generation** — worse than the 17x/172x this page recorded from the earlier machine, so the README's
+numbers understate it. Generation falls to **0.1 tokens per second**, roughly ten seconds per token,
+with 27,639 log lines mentioning OpenVINO.
+
+Two things this settles that the earlier run could not:
+
+* **Without the ICD the module does not open at all.** The loader places it (`with GPU module(s)
+  openvino`) and ggml reports two backends, with throughput identical to the CPU baseline. The
+  OpenVINO runtime archive does not ship `libOpenCL.so.1`; `ocl-icd-libopencl1` is what changes the
+  answer, and installing it is what makes the regression appear.
+* **The opt-in holds.** With the jar on the classpath and `net.ladenthin.llama.backend` unset, the
+  loader says `is on the classpath but not used`, two backends register and the machine runs at full
+  speed. This is the **first proof on hardware where OpenVINO could genuinely have taken over** — in
+  CI there is no Intel accelerator, so "nothing happened" proves nothing there.
+
 ### What to do about it
 
+
 Recorded in `TODO.md` as the owner's call, with a recommendation: **make the OpenVINO module opt-in**
 rather than additive. The premise of additive module jars is "a module that cannot help is skipped,
 one that can help is used" — OpenVINO satisfies neither on a CPU-only host. Failing that, the README