From 691faedd0d4b08186f9a988c1ab764e7f56ad600 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Sun, 11 Oct 2026 22:24:28 +0200 Subject: [PATCH] Declare the opt-in modules in natives.csv, so the smokes know what the loader leaves out Package JARs failed in run 38164077038 with: classpath: the loader did not put the openvino module jar of Linux/x86_64 in place. That is a consequence of making openvino opt-in, which I should have foreseen when I added OPT_IN_BACKENDS: smoke-natives-jars.sh asserts the loader line names every GPU module of the platform, read from natives.csv, and the list had no way to say that one of them is deliberately left out. The loader even printed the reason -- is on the classpath but not used: it is opt-in -- and nothing read it. Declared where every consumer already looks: an optin column in natives.csv. check-natives.py validates it (yes or no, only a module can be opt-in, and the rows of one backend cannot disagree, since the loader decides per backend) and holds LlamaLoader.OPT_IN_BACKENDS to it in both directions. A backend the list calls opt-in while the loader does not would ship a jar that takes over silently -- the measured OpenVINO regression the opt-in exists for -- and the reverse would leave a jar inert with no row saying why. The two smoke layers now read the same column with different conditions, which is the part worth getting right. The model-backed rows (smoke-lib.sh and its PowerShell twin) stop expecting an opt-in module, because the loader leaves it out and because OpenVINO's CPU plugin would otherwise take over a server round and run 172x slower. smoke-natives-jars.sh instead names every module of the platform through net.ladenthin.llama.backend, opt-in ones included: it only loads the library and exits, so nothing is at risk, and it becomes the only place in CI that loads an opt-in module at all -- the mechanism is exercised rather than merely respected. What this run did answer, for the record: all 54 build jobs green, both ROCm among them (Windows 3:08 h, Linux 2:57 h -- they are the three-hour pipeline, every other job finished inside an hour), and verify-native-deps.py reporting 99 native libraries checked, 0 violations. The SYCL run path and the removed libgomp dependency are therefore proven. Everything behind smoke-natives-jars.sh was skipped and remains unproven. 122 buildcheck tests, four new ones covering the column and both directions of the constant check. Both module lists and the property construction were simulated against the real natives.csv locally, including the macOS case with no module jars at all; check-run-scripts.py parses all 161 scripts. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01E2h8gXyyE5UeimkL9vQv9G --- .github/buildcheck/natives.py | 24 ++++++- .github/buildcheck/tests/test_natives.py | 87 ++++++++++++++++-------- .github/natives.csv | 48 ++++++------- .github/smoke-lib.sh | 7 +- .github/smoke-natives-jars.sh | 15 +++- .github/smoke-test-natives.ps1 | 3 + CLAUDE.md | 11 ++- TODO.md | 28 ++++++++ 8 files changed, 166 insertions(+), 57 deletions(-) diff --git a/.github/buildcheck/natives.py b/.github/buildcheck/natives.py index 75f1218bc..3eb8edfa6 100644 --- a/.github/buildcheck/natives.py +++ b/.github/buildcheck/natives.py @@ -12,7 +12,9 @@ uploads a smoke set llama-smoke- per smoke target, and every smoke target is launched by a job that downloads its set * LlamaLoader LIBRARY_BACKENDS / MODULE_BACKENDS know every backend of their kind (else a - jar ships and never loads) + jar ships and never loads), and OPT_IN_BACKENDS names exactly the backends + the optin column declares (else an opt-in jar takes over silently, or an + inert one has no row saying why) * CMakeLists.txt names exactly the backend directories of the list * nativedeps.ALLOWED holds an allowlist for every library directory (cpu, metal), and for no directory the list lacks @@ -137,6 +139,11 @@ def check_rows(natives): if not r["classifier"].startswith(backend(r) + "-") or r["platform"] not in ("yes", "no"): failures.append(f"natives.csv: row {r['classifier']} -- classifier must start with its " f"directory's backend '{backend(r)}', platform must be yes or no") + if r.get("optin") not in ("yes", "no"): + failures.append(f"natives.csv: row {r['classifier']} -- optin must be yes or no") + elif r["optin"] == "yes" and r.get("kind") != "module": + failures.append(f"natives.csv: row {r['classifier']} -- only a module can be opt-in: a library jar " + f"is the one jar that must load") if r.get("kind") not in KINDS: failures.append(f"natives.csv: row {r['classifier']} -- kind must be one of {KINDS}") elif r["kind"] == "library" and not re.fullmatch(r"(lib)?jllama\.(so|dll|dylib)", r["library"]): @@ -150,6 +157,11 @@ def check_rows(natives): failures.append(f"natives.csv: row {r['classifier']} -- a module cannot be a platform jar: it " f"holds no library") trees.setdefault(tree(r), []).append(r) + for b in sorted({backend(r) for r in natives}): + answers = {r.get("optin") for r in natives if backend(r) == b} + if len(answers) > 1: + failures.append(f"natives.csv: backend '{b}' is opt-in in one row and not in another " + f"({sorted(answers)}) -- the loader decides per backend, so the rows cannot differ") for t, group in sorted(trees.items()): if not any(r.get("kind") == "library" for r in group): failures.append(f"natives.csv: {t} has module jars but no library jar -- nothing would load them") @@ -238,6 +250,16 @@ def constant(name): block = re.search(name + r"\s*=(.*?);", loader_text, re.S) return set(re.findall(r'"([^"]+)"', block.group(1))) if block else set() failures = [] + # OPT_IN_BACKENDS is what makes a module jar inert until the property names it, so a backend the + # list declares opt-in while the loader does not would ship a jar that takes over silently -- + # which is the measured OpenVINO regression this column exists for -- and the other way round a + # jar would never load with no row saying why. + listed_optin = {backend(r) for r in natives if r.get("optin") == "yes"} + known_optin = constant("OPT_IN_BACKENDS") + failures += [f"LlamaLoader.OPT_IN_BACKENDS does not know '{b}', which natives.csv declares opt-in" + for b in sorted(listed_optin - known_optin)] + failures += [f"LlamaLoader.OPT_IN_BACKENDS names '{b}', which no row of natives.csv declares opt-in" + for b in sorted(known_optin - listed_optin)] for kind, const in (("library", "LIBRARY_BACKENDS"), ("module", "MODULE_BACKENDS")): listed = {backend(r) for r in natives if r["kind"] == kind} known = constant(const) diff --git a/.github/buildcheck/tests/test_natives.py b/.github/buildcheck/tests/test_natives.py index e557fe0c3..7e83c9b7e 100644 --- a/.github/buildcheck/tests/test_natives.py +++ b/.github/buildcheck/tests/test_natives.py @@ -11,15 +11,15 @@ from buildcheck.tests.helpers import REPO, upload, workflow_text CSV = """# a comment -classifier,directory,library,platform,kind -cpu-linux-x86-64,Linux/x86_64/cpu,libjllama.so,yes,library -cuda13-linux-x86-64,Linux/x86_64/cuda13,libggml-cuda.so,no,module -cpu-linux-s390x,Linux/s390x/cpu,libjllama.so,yes,library -cpu-windows-aarch64,Windows/aarch64/cpu,jllama.dll,yes,library -vulkan-windows-aarch64,Windows/aarch64/vulkan,ggml-vulkan.dll,no,module -cpu-android-aarch64,Linux-Android/aarch64/cpu,libjllama.so,no,library -opencl-android-aarch64,Linux-Android/aarch64/opencl,libggml-opencl.so,no,module -metal-macos-aarch64,Mac/aarch64/metal,libjllama.dylib,yes,library +classifier,directory,library,platform,kind,optin +cpu-linux-x86-64,Linux/x86_64/cpu,libjllama.so,yes,library,no +cuda13-linux-x86-64,Linux/x86_64/cuda13,libggml-cuda.so,no,module,no +cpu-linux-s390x,Linux/s390x/cpu,libjllama.so,yes,library,no +cpu-windows-aarch64,Windows/aarch64/cpu,jllama.dll,yes,library,no +vulkan-windows-aarch64,Windows/aarch64/vulkan,ggml-vulkan.dll,no,module,no +cpu-android-aarch64,Linux-Android/aarch64/cpu,libjllama.so,no,library,no +opencl-android-aarch64,Linux-Android/aarch64/opencl,libggml-opencl.so,no,module,yes +metal-macos-aarch64,Mac/aarch64/metal,libjllama.dylib,yes,library,no """ ROWS = natives.rows(CSV) @@ -67,7 +67,8 @@ class ListTest(unittest.TestCase): def test_rows_skip_comments(self): self.assertEqual(ROWS[0], {"classifier": "cpu-linux-x86-64", "directory": "Linux/x86_64/cpu", - "library": "libjllama.so", "platform": "yes", "kind": "library"}) + "library": "libjllama.so", "platform": "yes", "kind": "library", + "optin": "no"}) self.assertEqual(len(ROWS), 8) self.assertEqual([r["classifier"] for r in natives.modules(ROWS)], ["cuda13-linux-x86-64", "vulkan-windows-aarch64", "opencl-android-aarch64"]) @@ -86,10 +87,10 @@ def test_smoke_targets_are_the_desktop_library_platforms(self): def test_check_rows(self): self.assertEqual(natives.check_rows(ROWS), []) - bad = natives.rows("classifier,directory,library,platform,kind\n" - "cuda-linux-x86-64,Linux/x86_64/cuda13,x,no,module\n" - "cpu-a,A/b/cpu,libjllama.so,maybe,library\n" - "cpu-a,A/b/cpu,libjllama.so,no,thing\n") + bad = natives.rows("classifier,directory,library,platform,kind,optin\n" + "cuda-linux-x86-64,Linux/x86_64/cuda13,x,no,module,no\n" + "cpu-a,A/b/cpu,libjllama.so,maybe,library,no\n" + "cpu-a,A/b/cpu,libjllama.so,no,thing,no\n") failures = natives.check_rows(bad) self.assertEqual(len(failures), 7, failures) self.assertIn("twice", failures[0]) @@ -102,20 +103,20 @@ def test_check_rows(self): self.assertTrue(any(expected in f for f in failures), expected) def test_check_rows_refuses_a_module_without_a_library_and_colliding_file_names(self): - orphan = natives.rows("classifier,directory,library,platform,kind\n" - "vulkan-linux-aarch64,Linux/aarch64/vulkan,libggml-vulkan.so,no,module\n") + orphan = natives.rows("classifier,directory,library,platform,kind,optin\n" + "vulkan-linux-aarch64,Linux/aarch64/vulkan,libggml-vulkan.so,no,module,no\n") self.assertEqual(natives.check_rows(orphan), ["natives.csv: Linux/aarch64 has module jars but no library jar -- nothing would load them"]) - twice = natives.rows("classifier,directory,library,platform,kind\n" - "cpu-linux-x86-64,Linux/x86_64/cpu,libjllama.so,yes,library\n" - "sycl-linux-x86-64,Linux/x86_64/sycl,libggml-sycl.so,no,module\n" - "sycl-fp32-linux-x86-64,Linux/x86_64/sycl-fp32,libggml-sycl.so,no,module\n") + twice = natives.rows("classifier,directory,library,platform,kind,optin\n" + "cpu-linux-x86-64,Linux/x86_64/cpu,libjllama.so,yes,library,no\n" + "sycl-linux-x86-64,Linux/x86_64/sycl,libggml-sycl.so,no,module,no\n" + "sycl-fp32-linux-x86-64,Linux/x86_64/sycl-fp32,libggml-sycl.so,no,module,no\n") failures = natives.check_rows(twice) self.assertEqual(len(failures), 1, failures) self.assertIn("same name", failures[0]) - platform_module = natives.rows("classifier,directory,library,platform,kind\n" - "cpu-linux-x86-64,Linux/x86_64/cpu,libjllama.so,yes,library\n" - "cuda13-linux-x86-64,Linux/x86_64/cuda13,libggml-cuda.so,yes,module\n") + platform_module = natives.rows("classifier,directory,library,platform,kind,optin\n" + "cpu-linux-x86-64,Linux/x86_64/cpu,libjllama.so,yes,library,no\n" + "cuda13-linux-x86-64,Linux/x86_64/cuda13,libggml-cuda.so,yes,module,no\n") self.assertIn("cannot be a platform jar", natives.check_rows(platform_module)[0]) @@ -170,22 +171,54 @@ def test_a_matrix_launches_its_rows(self): ["publish.yml smoke runs (a job downloading llama-smoke-): missing windows-aarch64"]) def test_a_module_jar_without_a_smoke_target_is_refused(self): - rows = natives.rows(CSV + "vulkan-linux-s390x,Linux/s390x/vulkan,libggml-vulkan.so,no,module\n") + rows = natives.rows(CSV + "vulkan-linux-s390x,Linux/s390x/vulkan,libggml-vulkan.so,no,module,no\n") failures = natives.check_workflow(rows, jobs(rows=rows)) self.assertEqual(failures, ["natives.csv: the module jar vulkan-linux-s390x has no smoke target -- it would " "ship unlaunched"]) def test_loader(self): + # The fixture declares opencl opt-in (see CSV), so the loader text has to name it. text = ('List LIBRARY_BACKENDS = Arrays.asList("metal", "cpu");\n' - 'List MODULE_BACKENDS = Arrays.asList(\n "cuda13", "vulkan",\n "opencl");') + 'List MODULE_BACKENDS = Arrays.asList(\n "cuda13", "vulkan",\n "opencl");\n' + 'Set OPT_IN_BACKENDS = Collections.singleton("opencl");') self.assertEqual(natives.check_loader(ROWS, text), []) self.assertEqual(natives.check_loader(ROWS, text.replace('"cuda13", ', "")), ["LlamaLoader.MODULE_BACKENDS does not know 'cuda13' -- its jars would never load"]) - self.assertEqual(natives.check_loader(ROWS, text.replace('"opencl"', '"opencl", "msvc"')), - ["LlamaLoader.MODULE_BACKENDS names 'msvc', which no module jar of natives.csv ships"]) + self.assertEqual( + natives.check_loader(ROWS, text.replace('"opencl");' + chr(10), '"opencl", "msvc");' + chr(10))), + ["LlamaLoader.MODULE_BACKENDS names 'msvc', which no module jar of natives.csv ships"]) self.assertEqual(natives.check_loader(ROWS, text.replace('"metal", ', "")), ["LlamaLoader.LIBRARY_BACKENDS does not know 'metal' -- its jars would never load"]) + # The opt-in column and the loader constant have to agree in both directions. A backend the + # list calls opt-in while the loader does not would ship a jar that takes over silently -- + # the measured OpenVINO regression the column exists for -- and the other way round a jar + # would be inert with no row saying why. + self.assertEqual(natives.check_loader(ROWS, text.replace('Collections.singleton("opencl")', "emptySet()")), + ["LlamaLoader.OPT_IN_BACKENDS does not know 'opencl', which natives.csv declares opt-in"]) + self.assertEqual( + natives.check_loader(ROWS, text.replace('singleton("opencl")', 'asList("opencl", "vulkan")')), + ["LlamaLoader.OPT_IN_BACKENDS names 'vulkan', which no row of natives.csv declares opt-in"]) + + def test_check_rows_holds_the_optin_column(self): + missing = natives.rows("classifier,directory,library,platform,kind,optin\n" + "cpu-linux-x86-64,Linux/x86_64/cpu,libjllama.so,yes,library\n") + self.assertTrue(any("optin must be yes or no" in f for f in natives.check_rows(missing))) + + # Only a module can be opt-in: a library jar is the one jar that has to load. + library = natives.rows("classifier,directory,library,platform,kind,optin\n" + "cpu-linux-x86-64,Linux/x86_64/cpu,libjllama.so,yes,library,yes\n") + self.assertTrue(any("only a module can be opt-in" in f for f in natives.check_rows(library))) + + # Opt-in is decided per backend, so two jars of one backend cannot disagree -- one of them + # would be inert and the other not, which no reader would expect. + split = natives.rows("classifier,directory,library,platform,kind,optin\n" + "cpu-linux-x86-64,Linux/x86_64/cpu,libjllama.so,yes,library,no\n" + "cpu-windows-x86-64,Windows/x86_64/cpu,jllama.dll,yes,library,no\n" + "openvino-linux-x86-64,Linux/x86_64/openvino,libggml-openvino.so,no,module,yes\n" + "openvino-windows-x86-64,Windows/x86_64/openvino,ggml-openvino.dll,no,module,no\n") + self.assertTrue(any("is opt-in in one row and not in another" in f for f in natives.check_rows(split))) + def test_cmake_both_directions(self): text = "\n".join(f" set(JLLAMA_BACKEND {b})" for b in ("cuda13", "cpu", "vulkan", "opencl", "metal")) self.assertEqual(natives.check_cmake(ROWS, text), []) diff --git a/.github/natives.csv b/.github/natives.csv index ba5b3b8bc..a3855b6ca 100644 --- a/.github/natives.csv +++ b/.github/natives.csv @@ -19,27 +19,27 @@ # a build job uploading natives-, and, for a new backend, its name in CMakeLists.txt # and in LlamaLoader.LIBRARY_BACKENDS or MODULE_BACKENDS. .github/check-natives.py fails until all # of them agree. -classifier,directory,library,platform,kind -cpu-linux-x86-64,Linux/x86_64/cpu,libjllama.so,yes,library -cpu-linux-aarch64,Linux/aarch64/cpu,libjllama.so,yes,library -cpu-linux-s390x,Linux/s390x/cpu,libjllama.so,yes,library -cpu-android-aarch64,Linux-Android/aarch64/cpu,libjllama.so,no,library -cpu-android-x86-64,Linux-Android/x86_64/cpu,libjllama.so,no,library -cpu-windows-x86-64,Windows/x86_64/cpu,jllama.dll,yes,library -cpu-windows-aarch64,Windows/aarch64/cpu,jllama.dll,yes,library -metal-macos-aarch64,Mac/aarch64/metal,libjllama.dylib,yes,library -cuda13-linux-x86-64,Linux/x86_64/cuda13,libggml-cuda.so,no,module -cuda13-windows-x86-64,Windows/x86_64/cuda13,ggml-cuda.dll,no,module -rocm-linux-x86-64,Linux/x86_64/rocm,libggml-hip.so,no,module -rocm-windows-x86-64,Windows/x86_64/rocm,ggml-hip.dll,no,module -sycl-linux-x86-64,Linux/x86_64/sycl,libggml-sycl.so,no,module -sycl-windows-x86-64,Windows/x86_64/sycl,ggml-sycl.dll,no,module -vulkan-linux-aarch64,Linux/aarch64/vulkan,libggml-vulkan.so,no,module -vulkan-linux-x86-64,Linux/x86_64/vulkan,libggml-vulkan.so,no,module -vulkan-windows-aarch64,Windows/aarch64/vulkan,ggml-vulkan.dll,no,module -vulkan-windows-x86-64,Windows/x86_64/vulkan,ggml-vulkan.dll,no,module -opencl-android-aarch64,Linux-Android/aarch64/opencl,libggml-opencl.so,no,module -opencl-windows-aarch64,Windows/aarch64/opencl,ggml-opencl.dll,no,module -opencl-windows-x86-64,Windows/x86_64/opencl,ggml-opencl.dll,no,module -openvino-linux-x86-64,Linux/x86_64/openvino,libggml-openvino.so,no,module -openvino-windows-x86-64,Windows/x86_64/openvino,ggml-openvino.dll,no,module +classifier,directory,library,platform,kind,optin +cpu-linux-x86-64,Linux/x86_64/cpu,libjllama.so,yes,library,no +cpu-linux-aarch64,Linux/aarch64/cpu,libjllama.so,yes,library,no +cpu-linux-s390x,Linux/s390x/cpu,libjllama.so,yes,library,no +cpu-android-aarch64,Linux-Android/aarch64/cpu,libjllama.so,no,library,no +cpu-android-x86-64,Linux-Android/x86_64/cpu,libjllama.so,no,library,no +cpu-windows-x86-64,Windows/x86_64/cpu,jllama.dll,yes,library,no +cpu-windows-aarch64,Windows/aarch64/cpu,jllama.dll,yes,library,no +metal-macos-aarch64,Mac/aarch64/metal,libjllama.dylib,yes,library,no +cuda13-linux-x86-64,Linux/x86_64/cuda13,libggml-cuda.so,no,module,no +cuda13-windows-x86-64,Windows/x86_64/cuda13,ggml-cuda.dll,no,module,no +rocm-linux-x86-64,Linux/x86_64/rocm,libggml-hip.so,no,module,no +rocm-windows-x86-64,Windows/x86_64/rocm,ggml-hip.dll,no,module,no +sycl-linux-x86-64,Linux/x86_64/sycl,libggml-sycl.so,no,module,no +sycl-windows-x86-64,Windows/x86_64/sycl,ggml-sycl.dll,no,module,no +vulkan-linux-aarch64,Linux/aarch64/vulkan,libggml-vulkan.so,no,module,no +vulkan-linux-x86-64,Linux/x86_64/vulkan,libggml-vulkan.so,no,module,no +vulkan-windows-aarch64,Windows/aarch64/vulkan,ggml-vulkan.dll,no,module,no +vulkan-windows-x86-64,Windows/x86_64/vulkan,ggml-vulkan.dll,no,module,no +opencl-android-aarch64,Linux-Android/aarch64/opencl,libggml-opencl.so,no,module,no +opencl-windows-aarch64,Windows/aarch64/opencl,ggml-opencl.dll,no,module,no +opencl-windows-x86-64,Windows/x86_64/opencl,ggml-opencl.dll,no,module,no +openvino-linux-x86-64,Linux/x86_64/openvino,libggml-openvino.so,no,module,yes +openvino-windows-x86-64,Windows/x86_64/openvino,ggml-openvino.dll,no,module,yes diff --git a/.github/smoke-lib.sh b/.github/smoke-lib.sh index 06c9eadce..99b62b997 100644 --- a/.github/smoke-lib.sh +++ b/.github/smoke-lib.sh @@ -23,7 +23,12 @@ smoke_classpath() { # The GPU module backends whose jars are in the smoke set (from .github/natives.csv), one per line. smoke_set_modules() { local set_dir="$1" - grep -v -e '^#' -e '^classifier,' -e '^$' "$SMOKE_LIST" | tr -d '\r' | awk -F, '$5 == "module" { print $1 "," $2 }' \ + # $6 == "yes" is an opt-in module (natives.csv): its jar on the classpath does nothing until + # net.ladenthin.llama.backend names it, so the loader deliberately leaves it out and the line + # below must not expect it. smoke-natives-jars.sh names them instead, which is where the opt-in + # path itself is exercised. + grep -v -e '^#' -e '^classifier,' -e '^$' "$SMOKE_LIST" | tr -d '\r' \ + | awk -F, '$5 == "module" && $6 != "yes" { print $1 "," $2 }' \ | while IFS=, read -r classifier dir; do if find "$set_dir" -maxdepth 1 -name "llama-*-$classifier.jar" | grep -q .; then basename "$dir" diff --git a/.github/smoke-natives-jars.sh b/.github/smoke-natives-jars.sh index 2b93921ee..7e9ed2252 100755 --- a/.github/smoke-natives-jars.sh +++ b/.github/smoke-natives-jars.sh @@ -35,6 +35,17 @@ case "$(uname -s)-$(uname -m)" in esac expected_modules="$(grep -v -e '^#' -e '^classifier,' -e '^$' "$LIST" | tr -d '\r' \ | awk -F, -v t="$tree/" 'index($2, t) == 1 && $5 == "module" { sub(t, "", $2); print $2 }')" +# Every module of the platform, opt-in ones included, named through the property. This is the only +# place in CI that loads an opt-in module at all: the loader leaves it out by default (that is the +# point of the column), and the model-backed smoke rows must not have it, since OpenVINO's CPU +# plugin would take over and run 172x slower. Here nothing is inferred -- the library is loaded and +# the process exits -- so naming them proves both that the jars coexist and that the property +# enables what it names. +backend_property="" +if [ -n "$expected_modules" ]; then + backend_property="-Dnet.ladenthin.llama.backend=$(tr '\n' ',' <<< "$expected_modules" | sed 's/,$//')" + echo "enabling every module of $tree: $backend_property" +fi # The classes jar is the one without a classifier; every other jar but sources/javadoc/the fat jar # is a natives jar. @@ -64,5 +75,5 @@ run() { echo "OK ($label): $backend; $(grep 'native load smoke OK' <<< "$out")" } -run "classpath" -cp "$path" -run "module path" --module-path "$path" --add-modules ALL-MODULE-PATH +run "classpath" ${backend_property:+"$backend_property"} -cp "$path" +run "module path" ${backend_property:+"$backend_property"} --module-path "$path" --add-modules ALL-MODULE-PATH diff --git a/.github/smoke-test-natives.ps1 b/.github/smoke-test-natives.ps1 index 1a3334392..c5c9a2e5d 100644 --- a/.github/smoke-test-natives.ps1 +++ b/.github/smoke-test-natives.ps1 @@ -29,6 +29,9 @@ function Get-SetModules([string]$Dir) { if ($line -match '^#' -or $line -match '^classifier,' -or -not $line.Trim()) { continue } $cols = $line.Split(',') if ($cols[4] -ne 'module') { continue } + # An opt-in module (natives.csv column 6) is deliberately left out by the loader until + # net.ladenthin.llama.backend names it, so it is not expected in the loader line. + if ($cols.Length -gt 5 -and $cols[5] -eq 'yes') { continue } if (Get-ChildItem -Path $Dir -Filter "llama-*-$($cols[0]).jar" -File) { $modules += ($cols[1] -split '/')[-1] } diff --git a/CLAUDE.md b/CLAUDE.md index 77b3ef916..c85689193 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -43,6 +43,12 @@ artifact, classifier `--`, each holding exactly one directory and contributes a *device*, so llama.cpp prefers it and a host without an Intel accelerator drops from 490.4/34.5 to 29.4/**0.2** tokens per second, while without an OpenCL ICD loader (`libOpenCL.so.1`, which the OpenVINO archive does not ship) it does not open at all. + **The `optin` column of `natives.csv` is where that is declared**, and `check-natives.py` holds + `LlamaLoader.OPT_IN_BACKENDS` to it in both directions: a backend the list calls opt-in while the + loader does not would ship a jar that takes over silently, and the reverse would leave a jar inert + with no row saying why. The two smoke layers read the same column -- the model-backed rows do not + expect an opt-in module in the loader line, while `smoke-natives-jars.sh` names every module + through the property, which is the only place in CI that loads one. `net.ladenthin.llama.backend=,` is a **filter** over the module jars (`cpu` = none); a name that is no module backend, or whose jar is not on the classpath, fails the load with a message naming the jar to add. **A module jar needs the library jar of the same build**: every natives @@ -75,7 +81,7 @@ and 32-bit Windows (`cpu-windows-x86`, `msvc-windows-x86`), the MSVC jar (`msvc- still **built and tested**, see "Windows natives") and the second SYCL precision (`sycl-fp32`) went with the fat jars (5.2.0, before the release). -**The list is `.github/natives.csv`** (classifier, directory, library, platform yes/no, kind) — the +**The list is `.github/natives.csv`** (classifier, directory, library, platform yes/no, kind, optin yes/no) — the one place a natives jar is declared, in the same spirit as `models.csv`. Everything else either reads it or is checked against it: @@ -92,7 +98,8 @@ it or is checked against it: | `.github/merge-native-artifacts.sh` | reads the list: every listed `natives-*` artifact present, no other, each holding its library (or module) and nothing outside its directory, each with a `jllama-build.txt` naming a tag, a module directory holding nothing but its module and the `jllama-*.txt` lists, no path claimed twice; writes `jllama-extras.txt` (sibling files loaded before the library, e.g. OpenVINO's `OpenCL.dll` on Windows -- never the files a build's own `jllama-files.txt` names, which are only extracted; each of those must exist) | | `.github/package-smoke-sets.sh` | reads the list through `check-natives.py smoke-targets` / `smoke-set`: fails when a natives jar of a set was not built | | `.github/verify-native-deps.py` | every library of a directory: exact dependency allowlist per library directory (`cpu`/`metal`) plus the files next to it, and for the Android OpenCL module (bionic + `libOpenCL.so` + `libggml-base.so`); denylist for the other GPU modules, each of which must import `ggml-base` and, as ELF, reach it only through run path `$ORIGIN` (an Android library carries no run path -- bionic's search path is the app's native-library directory -- and only a build-tree one fails there); the manylinux_2_28 builds (both Linux CPU jars, CUDA) held to `GLIBC_2.28`; every Android library 16 KB page-aligned (Google Play). Runs in `package` and on the staged AAR libraries | -| `.github/smoke-natives-jars.sh` (`package` job) | loads the real jars: classes + all 23 natives jars at once, on the classpath **and** the module path; the loader line must name every GPU module of the runner's platform (from the list), so a module jar the loader skipped fails here | +| `.github/smoke-natives-jars.sh` (`package` job) | loads the real jars: classes + all 23 natives jars at once, on the classpath **and** the module path; it passes `net.ladenthin.llama.backend` naming **every** module of the runner's platform, opt-in ones included, and the loader line must then name each -- so a module jar the loader skipped fails here, and this is also the only place in CI that loads an opt-in module at all | +| `.github/smoke-lib.sh` / `smoke-test-natives.ps1` | the model-backed smoke rows expect every module of the set **except** the opt-in ones (`optin` column), because the loader deliberately leaves those out -- it is the same list, read with one more condition | **Adding a natives jar:** a row in `natives.csv`, the execution `check-natives.py pom` prints, a build job uploading `natives-` (`-DJLLAMA_MODULE_ONLY=ON -DGGML_=ON` for a GPU module), the diff --git a/TODO.md b/TODO.md index 9e7b107fb..79999166a 100644 --- a/TODO.md +++ b/TODO.md @@ -100,6 +100,34 @@ no network and no Android SDK, and it catches everything the AAR job can fail on the three Android ones red, so the questions behind `package` were still open; the dispatch after this fix is the one to read for them. +### Round 0, run 38164077038: `package` reached, and the opt-in decision had a loose end + +**What it answered, and it is the first time any of it has been answered:** all **54** build jobs +green, both ROCm among them (Windows 3:08 h, Linux 2:57 h -- they *are* the three-hour pipeline; every +other job was done inside an hour), and **`verify-native-deps.py`: 99 native libraries checked, 0 +violations.** So the SYCL run path is clean, and the Linux jars no longer declare `libgomp.so.1` +without anything noticing. + +**What it found:** `smoke-natives-jars.sh` failed with +`classpath: the loader did not put the openvino module jar of Linux/x86_64 in place` -- a consequence +of making `openvino` opt-in, which I should have foreseen when I added `OPT_IN_BACKENDS`: that script +asserts the loader line names **every** GPU module of the platform, read from `natives.csv`, and the +list had no way to say that one of them is deliberately left out. The loader even printed the reason +("is on the classpath but not used: it is opt-in"); nothing read it. + +Fixed by declaring it where every consumer already looks: an **`optin` column in `natives.csv`**. +`check-natives.py` now holds `LlamaLoader.OPT_IN_BACKENDS` to that column in both directions, the +model-backed smoke rows stop expecting an opt-in module (bash and PowerShell), and +`smoke-natives-jars.sh` names every module through the property instead -- which makes it **the only +place in CI that loads an opt-in module at all**, so the mechanism is exercised rather than merely +respected. 122 buildcheck tests, four of them new; both module lists and the property construction +were simulated against the real CSV locally. + +**Still unanswered, and only a run can answer it:** everything behind `smoke-natives-jars.sh` -- +the five smoke sets, the four `smoke-natives` rows with the RPC smoke on aarch64, the macOS smoke and +the agent smoke were all **skipped**, as they were in every run before. Two emulator jobs stayed red +in this run by construction (their fix landed afterwards, #526). + ### Hand-over from the bump session, 2026-10-11 What that session did: the b11538-to-b11556 bump (#513 -- every patch re-verified, two history rows,