Repository navigation
Round 2 measured: the T4 abort is gone, and what loading the natives costs - #530
Merged
Merged
Conversation
added 2 commits
October 12, 2026 09:44
…costs All three round-2 questions answered against the PUBLISHED 5.2.0-SNAPSHOT jars rather than a build tree, so the numbers are what a consumer resolving from Maven gets. The CUDA finding is resolved and re-measured on the same hardware: same AMI, same driver 595.91.07, same Tesla T4, only the module changed. Three exit-134 aborts became three clean runs, and the GPU delivers 20.6x prompt processing and 7.0x generation over the CPU baseline on that machine. That -ngl 0 runs again closes the case, since it was the most absurd of the three. cuobjdump on the published module shows native code for sm_75, sm_80, sm_86, sm_89, sm_90, sm_100, sm_120a and sm_121a plus PTX for sm_121a, so Turing, A100 and Hopper no longer depend on the driver and anything newer than Blackwell still has a fallback. The a suffixes are upstream's own rewrite of every 12x architecture, not a defect in the list this project passes. A method note went in with it, because it nearly produced a wrong conclusion: a first scan reported no PTX because it searched for compute_, while cuobjdump names those files sm_121a.ptx. OpenVINO was re-measured on an Intel Sapphire Rapids instance and the README's figures turn out too kind: 21.7x slower prompt processing and 336x slower generation against the baseline of the same machine, not 17x/172x. Two things the earlier run could not settle are settled now -- without an OpenCL ICD the module does not open at all, and with the jar on the classpath but the property unset the loader leaves it out and the machine runs at full speed. That last one is the only hardware proof the opt-in decision has, because CI has no Intel accelerator and nothing happening there proves nothing. New page, native-loading.md: extracting 18 files and 39 MB costs about 810 ms per JVM start on Windows and the reuse removes it; the directory key visibly includes the module set; two JVMs of one build run concurrently; and the sharpest case, a sweep against a backdated directory while an RpcServer holds it, leaves exactly the five files that JVM still had open -- the library, the two pre-loaded ggml libraries, the one CPU module ggml kept and ggml-rpc -- while the 13 it had scored and closed go and are re-extracted on the next start. The server kept running throughout. The page also records which container base images can load the Linux jar now that libgomp is gone, including the musl image that must fail and does so by name. The harness page gains a tenth invariant from two wrong conclusions this session: the harness is the first suspect, not the artifact. A Windows classpath mangled by the POSIX shell made the loader correctly report no natives jar, and a local probe called five SDK layouts broken because chmod +x does not stick on that filesystem. Reproduce the expected case first; if even that fails, the harness is wrong. No private detail reaches these pages: no bucket, account, instance id, AMI of the account's own making or presigned URL. The account-specific record is in the private workspace. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E2h8gXyyE5UeimkL9vQv9G
Two numbers in the README were wrong, and one paragraph described a state that no longer exists. The CUDA paragraph was written before the T4 was re-measured: it told a reader that the driver must be new enough for CUDA 13.4, full stop. That is now only true for a GPU newer than the shipped set. The module compiles in sm_75 (T4, RTX 20xx), sm_80 (A100), sm_86, sm_89, sm_90 (H100), sm_100 and Blackwell, so those do not depend on the driver at all; only something newer falls back to PTX, and only there can a too-old driver still end the process. The paragraph now says that, and names what the measurement cost to learn: the previous module aborted the JVM in all three configurations on a Tesla T4 with driver 595.91.07, while the same AMI, driver and GPU with native sm_75 code runs cleanly at 20.6x prompt processing and 7.0x generation over the CPU baseline. OpenVINO's cost was understated. Re-measured on an Intel Sapphire Rapids instance it is 21.7x slower prompt processing and 336x slower generation against the baseline of the same machine, not 17x/172x -- 481.3/33.6 down to 22.2/0.1 tokens per second, about ten seconds per token. The ICD package name is in the text now too, since that is what a consumer has to install or deliberately not install. TODO.md closes round 2 with what each item answered rather than deleting it: the T4 abort and the architecture listing, the Windows extraction measurement including the five files a running JVM keeps locked, and the container matrix with the musl row that must fail and does so by name. Both write-ups are linked; no account detail is in any of it. check-natives.py 0 disagreements, check-patches.py 0 problems, 122 buildcheck tests green, and a search for bucket names, account ids, instance ids, AMIs and presigned URLs over README.md, TODO.md and docs/benchmarks finds nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E2h8gXyyE5UeimkL9vQv9G
bernardladenthin
had a problem deploying
to
startgate
October 12, 2026 07:49 — with 
GitHub Actions
Error
GitHub Actions
Error
bernardladenthin
had a problem deploying
to
maven-central
October 12, 2026 07:49 — with
GitHub Actions
Failure
GitHub Actions
Failure
bernardladenthin
had a problem deploying
to
maven-central
October 12, 2026 07:49 — with
GitHub Actions
Failure
GitHub Actions
Failure
|
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
All three round-2 questions answered against the published 5.2.0-SNAPSHOT jars
(
20261012.035453-12) rather than a build tree — so these are the numbers a consumer resolving fromMaven gets. Two AWS boots, under 0.08 USD together, both terminated, four regions swept afterwards.
1 · The T4 abort is gone
Same AMI, same driver 595.91.07, same Tesla T4. Only the module changed.
sm_75cubin)-ngl 99cuda13forced,-ngl 99cuda13forced,-ngl 0vulkan(control)cpu(baseline)Three aborts became three clean runs, and the GPU gives 20.6× prompt processing / 7.0× generation
over the CPU baseline. That
-ngl 0runs again closes it: that was the most absurd of the three,since not one layer was on the GPU.
cuobjdumpon the published module: native code forsm_75sm_80sm_86sm_89sm_90sm_100sm_120asm_121a, plus PTX forsm_121a. Theasuffixes are upstream's own rewrite of every12xarchitecture, not a defect in the list this project passes.2 · New page:
docs/benchmarks/native-loading.mdExtraction costs ~810 ms per JVM start on Windows (cold ~2175 ms, warm ~1365 ms; 18 files, 39 MB)
and the reuse removes it — per start, so also per surefire fork. The directory key visibly carries
the module set. Two JVMs of one build run concurrently.
The sharpest case: with an
RpcServerholding the library and the directory backdated pastSTALE_EXTRACTION_AGE, a sweeper left exactly the five files that JVM still had open — thelibrary, the two pre-loaded ggml libraries, the one CPU module ggml kept,
ggml-rpc— while the 13 ithad scored and closed went, and were re-extracted on the next start. The server kept running.
Container matrix, now that
libgompis gone from the artifact: distroless andtemurin:21-jreload;
temurin:21-jre-alpineis refused by name (os.name=Linux-Musl). That row is in the matrixon purpose — a measurement that only admits successes cannot show where the boundary is.
3 · Corrections the measurements forced
481.3/33.6 down to 22.2/0.1 tokens per second — about ten seconds per token. The README's figures
understated it.
And the README's CUDA paragraph described a state that no longer exists: it told a reader the driver
must be new enough for CUDA 13.4, full stop. That is now true only for a GPU newer than the shipped
set.
Also settled, and only measurable on this hardware: with the OpenVINO jar on the classpath and
net.ladenthin.llama.backendunset, the loader saysis on the classpath but not usedand the machineruns at full speed. That is the only hardware proof the opt-in decision has — CI has no Intel
accelerator, so nothing happening there proves nothing.
A tenth harness invariant
Two things this session looked like product defects and were neither: a Windows classpath mangled by
the POSIX shell (the loader then correctly reported no natives jar), and a local probe that called all
five SDK layouts broken because
chmod +xdoes not stick on that filesystem. The harness is thefirst suspect, not the artifact — reproduce the case that is expected to work first.
Scope and privacy
docs/benchmarks/(one new page, three updated),README.mdandTODO.md. The CUDA incident'srecord stays in place with the resolution written above it — what happened remains readable.
No account detail reaches these pages: a search over all of them for bucket names, account ids,
instance ids, AMIs and presigned URLs finds nothing. The account-specific record is committed in the
private workspace.
check-natives.py0 disagreements,check-patches.py0 problems, 122 buildcheck tests green.🤖 Generated with Claude Code
https://claude.ai/code/session_01E2h8gXyyE5UeimkL9vQv9G