diff --git a/.agents/adding-backends.md b/.agents/adding-backends.md index a7880a17b400..ac169b534f5d 100644 --- a/.agents/adding-backends.md +++ b/.agents/adding-backends.md @@ -34,7 +34,7 @@ The build matrix is data-only YAML at `.github/backend-matrix.yml` (not inside ` **Without an entry here no image is ever built or pushed, and the gallery entry in `backend/index.yaml` will point at a tag that does not exist.** The `dockerfile:` field must point at `./backend/Dockerfile.` matching the language bucket from step 1 (e.g. `Dockerfile.python`, `Dockerfile.golang`, `Dockerfile.rust`). The `tag-suffix` must match the `uri:` in the corresponding `backend/index.yaml` image entry exactly. -**Path-filter registration — REQUIRED for any new dockerfile suffix.** This is the single most common omission, because it has no effect on the PR that adds the backend (when no prior path filter could catch it anyway) — it only breaks the *next* PR that touches your backend's directory, which then gets zero CI jobs and looks broken for unrelated reasons. Edit `scripts/lib/backend-filter.mjs:inferBackendPath` and add a branch BEFORE the more-generic suffixes: +**`scripts/changed-backends.js` registration — REQUIRED for any new dockerfile suffix.** This is the single most common omission, because it has no effect on the PR that adds the backend (when no prior path filter could catch it anyway) — it only breaks the *next* PR that touches your backend's directory, which then gets zero CI jobs and looks broken for unrelated reasons. Edit `scripts/changed-backends.js:inferBackendPath` and add a branch BEFORE the more-generic suffixes: ```js if (item.dockerfile.endsWith("")) { @@ -54,9 +54,7 @@ for (const e of m.include.filter(e => e.backend === '')) { }" ``` -A quick way to find the right insertion point: `grep -n 'item.dockerfile.endsWith' scripts/lib/backend-filter.mjs`. - -If your backend consumes a *shared* build input that lives outside its own directory (a new script under `scripts/build/`, a new file copied into every image), add a rule to `SHARED_BUILD_INPUTS` in the same file — the per-backend prefix match cannot see those, and a miss ships your change to no image at all. See `scripts/lib/backend-filter_test.mjs` for the pattern; `make test-ci-scripts` runs it. +A quick way to find the right insertion point: `grep -n 'item.dockerfile.endsWith' scripts/changed-backends.js`. **`bump_deps.yaml` registration — REQUIRED for any backend pinning an upstream commit.** If your backend's Makefile has a `*_VERSION?=` pin to a third-party repo, the daily auto-bump bot at `.github/workflows/bump_deps.yaml` won't notice it unless you register the backend in its matrix. The bot runs `.github/bump_deps.sh` which `grep`s for `^$VAR?=` in the Makefile you list — so the pin MUST live in the Makefile (not in a separate shell script). The bump for ds4 (#9761) had to walk this back because the original landed the pin in `prepare.sh`, which the bot can't see. Pattern (for `antirez/ds4`): @@ -117,7 +115,7 @@ Wiring a backend into `includeDarwin:` is more than the matrix entry: 1. **`includeDarwin:` entry** — `tag-suffix: "-metal-darwin-arm64-"`, `build-type: "metal"`, `lang: "go"` for go+ggml backends; omit `build-type` for the bespoke C++ ones (llama-cpp / ds4 / privacy-filter). Match an existing entry of the same shape. 2. **`backend/index.yaml`** — add `metal:` to the backend's `capabilities` map (main and `-development`) and concrete `metal-` / `metal--development` image entries pointing at the `-metal-darwin-arm64-` images. -3. **C/C++ backends only** — add an `inferBackendPathDarwin` case in `scripts/lib/backend-filter.mjs` returning `backend/cpp//` (the generic fallthrough assumes `backend//`, which is wrong for a C++ source tree driven with `lang: go`), and give `run.sh` a Darwin branch that exports `DYLD_LIBRARY_PATH` instead of `LD_LIBRARY_PATH`. If the build is bespoke (single `grpc-server` + dylib bundling), model it on `scripts/build/ds4-darwin.sh` and add a `backends/-darwin` make target plus a gated step in `.github/workflows/backend_build_darwin.yml`. +3. **C/C++ backends only** — add an `inferBackendPathDarwin` case in `scripts/changed-backends.js` returning `backend/cpp//` (the generic fallthrough assumes `backend//`, which is wrong for a C++ source tree driven with `lang: go`), and give `run.sh` a Darwin branch that exports `DYLD_LIBRARY_PATH` instead of `LD_LIBRARY_PATH`. If the build is bespoke (single `grpc-server` + dylib bundling), model it on `scripts/build/ds4-darwin.sh` and add a `backends/-darwin` make target plus a gated step in `.github/workflows/backend_build_darwin.yml`. 4. **C++ proto gotcha** — if the backend compiles the generated gRPC/protobuf in a separate CMake target (e.g. `hw_grpc_proto`), that target must link `protobuf::libprotobuf` + `gRPC::grpc++` so the Homebrew include dirs propagate; otherwise macOS fails with `google/protobuf/runtime_version.h not found` (Linux hides this because apt headers sit in `/usr/include`). The CI path filter only builds a backend on a PR when a file under its directory changes, so a darwin-only YAML edit builds nothing — touch a file under `backend///` (a one-line comment is enough) in the same PR. @@ -218,69 +216,6 @@ docker-build-backends: ... docker-build- - If the backend is in `backend/python//` but uses `.` as context in the workflow file, use `.` context - Check similar backends to determine the correct context -## Engine preference for gallery model variants - -A gallery entry can declare `variants`, alternative builds of the same weights, -and LocalAI picks one per host: it drops builds whose backend cannot run here or -that do not fit memory, then ranks the survivors by **engine preference -first, serving feature second, size third** (`SelectVariant` in -`core/gallery/resolve_variant.go`). - -Ask whether your backend should outrank another one on some hardware. If it -should, add it to `engineNamePreferenceRules` in `pkg/system/capabilities.go`, -best engine first for that capability: - -```go - {Nvidia, []string{engineVLLM, engineSGLang, engineLlamaCpp}}, -+ {Nvidia, []string{engineVLLM, engineSGLang, engineMyEngine, engineLlamaCpp}}, -``` - -That is the ENGINE NAME table, matched as a substring of a gallery entry's -`backend:` value. Two sibling tables in the same file speak different -vocabularies and are matched against different things: - -| Table | Vocabulary | Matched against | Consumer | -|-------|-----------|-----------------|----------| -| `backendBuildTagPreferenceRules` | build tags (`cuda`, `rocm`, `metal`) | installed build directory names, as a substring | alias resolution in `ListSystemBackends` | -| `engineNamePreferenceRules` | engine names (`vllm`, `llama-cpp`, `mlx`) | a gallery entry's `backend:`, as a substring | gallery variant ranking | -| `servingFeaturePreferenceTokens` | serving features (`dflash`, `mtp`) | a gallery entry's `tags:`, compared whole and case-insensitively, and nothing else | gallery variant ranking, one rank below the engine | - -**Putting a token in the wrong table matches nothing and does not error**: every -candidate scores equal and the next sort key decides, so the preference silently -stops existing. The block comment above all three tables spells the contract out. - -The serving feature table is the odd one: it is not keyed by capability, because -no hardware prefers a plain build over an equivalent faster build of the same -weights. It reads a declared tag and nothing else. The entry name was the -original signal and is gone: a naming convention is not a contract, and names -are author-supplied free text where a short marker like `mtp` turns up inside -unrelated words or on weights whose entry enables nothing. -`overrides.options` was rejected for the mirror-image reason: `spec_type:` is -llama.cpp's config vocabulary, whereas a cross-backend ranking decision must -work the same for `ds4`'s `mtp_path:` and `sglang`'s `speculative_algorithm:`. - -**If your backend can serve the same weights faster** (speculative decoding, -multi-token prediction), say so in the docs for its gallery entries so curators -tag them: the tagging rule and the per-backend evidence table live in -[adding-gallery-models.md](adding-gallery-models.md). A backend never needs to -appear in the token table itself; it ranks builds, not engines. - -Leaving your backend out is a valid choice when no ordering can be justified for -it. It then ranks below every known engine and selection falls back to size, -which is the behaviour that predates preference. - -**Leaving a whole capability out is not.** A missing row gives that host an -empty preference list, so size alone decides among everything that survives the -filters, and the filter will not save you: `IsBackendCompatible` derives hardware -support from the engine NAME, so `vllm` and `sglang` carry no darwin, cuda, rocm -or sycl token and are never dropped on a host with no GPU. That is why `default` -(no usable accelerator, including a GPU under the 4 GiB VRAM floor) and -`darwin-x86` both have rows putting `llama-cpp` first. Every capability -`getSystemCapabilities()` can return needs a row unless every engine really is -equally at home there. When you add one, enumerate the engines you are demoting -rather than relying on them falling through unmatched: unmatched engines all tie -with each other, so size decides among them. - ## Documenting the backend (README + docs) A backend is not "added" until it is discoverable. Update the user-facing docs: @@ -308,7 +243,7 @@ After adding a new backend, verify: - [ ] Backend directory structure is complete with all necessary files - [ ] Build configurations added to `.github/backend-matrix.yml` for all desired platforms (per-arch entries with `platform-tag` for multi-arch; `builder-base-image` for llama-cpp / ik-llama-cpp / turboquant) -- [ ] **OS coverage considered**: added to `includeDarwin:` (macOS/Apple Silicon) if the backend can build there — with the `backend/index.yaml` `metal:` capability + `metal-` image entries, a `run.sh` Darwin/DYLD branch and `inferBackendPathDarwin` case (in `scripts/lib/backend-filter.mjs`) for C++ backends — or the PR explains why an OS is unsupported. Do not ship Linux-only by default. +- [ ] **OS coverage considered**: added to `includeDarwin:` (macOS/Apple Silicon) if the backend can build there — with the `backend/index.yaml` `metal:` capability + `metal-` image entries, a `run.sh` Darwin/DYLD branch and `inferBackendPathDarwin` case for C++ backends — or the PR explains why an OS is unsupported. Do not ship Linux-only by default. - [ ] Meta definition added to `backend/index.yaml` in the `## metas` section - [ ] Image entries added to `backend/index.yaml` for all build variants (latest + development) - [ ] Tag suffixes match between workflow file and index.yaml @@ -317,7 +252,6 @@ After adding a new backend, verify: - [ ] No Makefile syntax errors (check with linter) - [ ] Follows the same pattern as similar backends (e.g., if it's a transcription backend, follow `faster-whisper` pattern) - [ ] **`Load` validates its input and refuses models it can't serve.** When a model config has no explicit `backend:`, the model loader greedily probes *every* installed backend with the model's name and binds to the first `Load` that succeeds — an accept-anything `Load` will capture arbitrary LLMs (issue #9287). Backends that load a real artefact get this for free (the load fails); backends with no artefact must gate on the name: `opus` accepts only its own name (or none), `local-store` requires the `store.NamespacePrefix` namespace marker sent by `core/backend/stores.go`. -- [ ] **Gallery variant ranking considered**: if this backend should be preferred over another on some hardware, it is listed in `engineNamePreferenceRules` (NOT `backendBuildTagPreferenceRules`, NOT `servingFeaturePreferenceTokens`) in `pkg/system/capabilities.go`. A missing entry silently ranks it last and lets the next sort key decide. - [ ] Documented: added to the category list in `docs/content/features/backends.md` (and any new endpoint/realtime capability documented under `docs/content/`) - [ ] If it is an in-house native C/C++/GGML engine, added to the maintained-engines table in the top-level `README.md` diff --git a/.agents/adding-gallery-models.md b/.agents/adding-gallery-models.md index 51221608f66a..fe5a57e87177 100644 --- a/.agents/adding-gallery-models.md +++ b/.agents/adding-gallery-models.md @@ -91,108 +91,6 @@ To add a variant (e.g., different quantization), use YAML merge: uri: huggingface:////-Q8_0.gguf ``` -## Offering several builds of one model (`variants`) - -When the same model is published in more than one quantization, or is also -servable by another engine, add each build as its own ordinary gallery entry and -then point one of them at the others with `variants`: - -```yaml -- !!merge <<: *chatml - name: "nanbeige4.1-3b-q4" - # ... the usual urls / overrides / files for the Q4 build ... - variants: - - model: nanbeige4.1-3b-q8 -``` - -Rules: - -- The declaring entry is a **complete, normal entry**. It keeps its own - `files`/`overrides` and stays installable on every host and by every older - LocalAI release, which simply ignore `variants`. -- A variant references another gallery entry **by name**. That entry must exist - and must not declare `variants` of its own. -- **A referenced entry keeps its own gallery row by default.** It is hidden only - in the collapsed listing (`collapse_variants=true`, which the web UI requests - by default), where the declaring entry stands in for it. Searching there still - matches the referenced entry and answers with the entry declaring it, so - referencing an entry never makes it unfindable; turning the collapse off - returns it under its own name. -- **Order carries no meaning.** Do not try to encode a preference; write the - list in whatever order reads best. -- **A variant may be smaller than the declaring entry.** Offering a downgrade - for small hosts is a normal shape: the declaring entry's own build competes - like every other candidate, so a large host keeps the large build. -- **Do not describe hardware.** At install time LocalAI drops variants whose - backend cannot run on the host, then drops those that do not fit available - memory. The declaring entry's own build is exempt from both filters, so - selection always terminates on something installable. Sizes are measured live - from the weights and cached, so nothing has to be written down. -- **Engine preference outranks size.** Among the builds that survive the - filters, the host's preferred engine wins first and only then does the larger - footprint win. On NVIDIA a vLLM build beats a larger llama.cpp one; on Apple - silicon an MLX build beats a larger GGUF one; on a host with no preference for - either engine the larger build wins, since a bigger footprint is a higher - quality quantization of the same weights. Predict what a user gets by asking - which engine the host prefers before asking which build is biggest. The - per-capability order lives in `engineNamePreferenceRules` - (`pkg/system/capabilities.go`); see - [adding-backends.md](adding-backends.md) for how a backend gets into it. -- **Serving feature preference sits between engine and size.** Among builds on - an equally preferred engine, one that speculates or predicts several tokens - per step beats the plain build of the same weights, because it answers faster - for the same output: a `dflash` build beats an `mtp` one, and either beats a - plain build. The order lives in `servingFeaturePreferenceTokens` - (`pkg/system/capabilities.go`) and is matched against the entry's `tags:` and - **nothing else**: not the entry name, not `overrides.options`. See - [the tagging rule](#the-dflash--mtp-tagging-rule) below. Engine deliberately - outranks it: a serving feature makes the right engine faster, it does not make - a wrong engine right. Fit still outranks both, so a drafter pairing (strictly - larger than the plain build, since it ships a drafter alongside it) is dropped - on a host too small for it before this order is ever consulted. -- A variant is nothing but a name; there is no per-variant memory field. When - the measured size for a build is wrong, correct it on the referenced entry by - setting that entry's own `size:` (e.g. `size: "20GiB"`). The estimator prefers - a declared size over its own guesswork, so the fix applies everywhere the size - is shown or compared rather than only to variant selection. - -Users can override the automatic choice with `variant` on `POST /models/apply`, -`local-ai models install --variant`, or the `install_model` MCP tool. See -`docs/content/features/model-gallery.md`. - -The gallery lint specs live in `core/gallery`, so run that suite after adding a -`variants` list. - -### The `dflash` / `mtp` tagging rule - -**Tag an entry `dflash` or `mtp` when the entry actually configures that -feature. Variant ranking reads the tag and nothing else.** - -Decide by looking at what the entry configures, in whatever vocabulary its -backend uses: - -| Backend | Configures the feature when it declares | -|---------|------------------------------------------| -| `llama-cpp` | `overrides.options` contains `spec_type:draft-dflash` or `spec_type:draft-mtp` | -| `ds4` | `overrides.options` contains `mtp_path:` / `mtp_draft:` | -| `sglang` | the referenced `gallery/*.yaml` sets `speculative_algorithm:` | - -That check is curation-time only. `spec_type` is llama.cpp's config vocabulary, -and a cross-backend ranking decision must not depend on one backend's option -syntax, which is precisely why the ranker reads the tag instead of the options. - -Two mistakes the rule exists to prevent: - -- **Weights that carry the heads are not an entry that enables them.** The - NVFP4 GGUF entries ship MTP-bearing weights but set only `use_jinja:true`, so - they enable no speculative decoding and must NOT be tagged. Tagging them wins - them the feature axis without being any faster. -- **A name is not a declaration.** An entry whose name spells `-mtp` while - configuring nothing gets no tag, and an entry that configures the feature is - tagged even when its name says nothing (`hy3`, `glm-5.2`). Ranking never reads - the name, so an untagged build that does enable the feature is simply ranked - as plain rather than promoted on a marker nobody meant. - ## Available template configs Look at existing `.yaml` files in `gallery/` to find the right prompt template for your model architecture: diff --git a/.agents/ci-caching.md b/.agents/ci-caching.md index 17cc6001d908..6193b494a238 100644 --- a/.agents/ci-caching.md +++ b/.agents/ci-caching.md @@ -114,24 +114,6 @@ Both `backend.yml` (push) and `backend_pr.yml` (PR) generate their matrix dynami - **Tag pushes**: `FORCE_ALL=true` is set from the workflow side (`startsWith(github.ref, 'refs/tags/')`) — releases rebuild every backend regardless of diff. - **Schedule / `workflow_dispatch`**: no `event.before`, falls through to "run everything" automatically. -### Shared build inputs - -The per-backend prefix match only sees files under a backend's own directory, so a change to shared build infrastructure would rebuild *nothing* — an empty matrix, every job green, and the change reaching no image. That silently un-shipped PR #10946 (a partial-cuDNN packaging fix in `scripts/build/package-gpu-libs.sh`), which merged 1h48m after the weekly cron and so sat unbuilt for a week. - -`SHARED_BUILD_INPUTS` in `scripts/lib/backend-filter.mjs` closes that hole. Each rule maps a shared path to the narrowest set of matrix entries it can honestly invalidate, since a full matrix is 417 Linux + 56 Darwin builds: - -| Changed path | Rebuilds | -|---|---| -| `backend/backend.proto` | everything (all languages compile or copy it) | -| `backend/Dockerfile.` | the Linux entries whose `dockerfile:` names it | -| `backend/python/common/` | Python, Linux + Darwin | -| `scripts/build/package-gpu-libs.sh` | Python, Linux only | -| `scripts/build/-darwin.sh` | the Darwin entries that build target routes to | -| `.github/workflows/backend_build[_darwin].yml` | everything on that OS | -| anything else under `scripts/build/` (except `*_test.sh`) | everything — conservative default for unclassified packaging inputs | - -Deliberately excluded: `backend/index.yaml` (gallery metadata, never enters an image), `.github/backend-matrix.yml` (adding a backend would rebuild all of them), `backend/Dockerfile.base-grpc-builder` (owned by `base-images.yml`), and the root `Makefile` (touched in ~11% of commits, and its backend-relevant edits arrive alongside the backend directory anyway). `make test-ci-scripts` pins all of this. - The Sunday 06:00 UTC cron on `backend.yml` exists specifically because path filtering can leave Python backends frozen on stale wheels. `DEPS_REFRESH` (below) only fires when the build actually runs, so an untouched Python backend would never re-resolve its unpinned deps. The weekly cron is the safety net. ## The `DEPS_REFRESH` cache-buster (Python backends) diff --git a/.agents/coding-style.md b/.agents/coding-style.md index ddee45ae28ec..da10c61f0a45 100644 --- a/.agents/coding-style.md +++ b/.agents/coding-style.md @@ -65,7 +65,6 @@ This is enforced by `forbidigo` (see `.golangci.yml`): `http.DefaultClient` and The project documentation is located in `docs/content`. When adding new features or changing existing functionality, it is crucial to update the documentation to reflect these changes. This helps users understand how to use the new capabilities and ensures the documentation stays relevant. -- **Docs-with-code rule**: When you change user-facing behavior (API endpoints, CLI flags, config keys, or features), update the corresponding page under `docs/content/` in the SAME change, not as a follow-up. A user-facing change without a matching docs update is incomplete. The PR template carries a checklist item for this. - **Feature Documentation**: If you add a new feature (like a new backend or API endpoint), create a new markdown file in `docs/content/features/` explaining what it is, how to configure it, and how to use it. - **Configuration**: If you modify configuration options, update the relevant sections in `docs/content/`. - **Examples**: providing concrete examples (like YAML configuration blocks) is highly encouraged to help users get started quickly. diff --git a/.docker/bonsai-compile.sh b/.docker/bonsai-compile.sh deleted file mode 100755 index 38439439d541..000000000000 --- a/.docker/bonsai-compile.sh +++ /dev/null @@ -1,39 +0,0 @@ -#!/usr/bin/env bash -# Shared compile logic for backend/Dockerfile.bonsai. -# Sourced (via bind mount) from both builder-fromsource and builder-prebuilt stages. - -set -euxo pipefail - -export CCACHE_DIR=/root/.ccache -ccache --max-size=5G || true -ccache -z || true - -export CMAKE_ARGS="${CMAKE_ARGS:-} -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_CUDA_COMPILER_LAUNCHER=ccache" - -if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then - CUDA_ARCH_ESC="${CUDA_DOCKER_ARCH//;/\\;}" - export CMAKE_ARGS="${CMAKE_ARGS} -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH_ESC}" - echo "CMAKE_ARGS(env) = ${CMAKE_ARGS}" - rm -rf /LocalAI/backend/cpp/bonsai-*-build -fi - -cd /LocalAI/backend/cpp/bonsai - -if [ -z "${BUILD_TYPE:-}" ]; then - # Pure CPU image: one ggml CPU_ALL_VARIANTS build replaces the per-microarch binaries. - # arm64: the armv9.2 SME variants need gcc-14 (gcc-13 rejects +sme). - if [ "${TARGETARCH}" = "arm64" ]; then - apt-get update -qq && apt-get install -y -qq gcc-14 g++-14 - export CC=gcc-14 CXX=g++-14 - fi - make bonsai-cpu-all -else - # GPU build (cublas/hipblas/sycl/vulkan/...): single fallback CPU build, the accelerator - # does the compute. Keeps the GPU compile from also building the CPU variant matrix and - # avoids the gcc-14 apt step on GPU base images such as nvidia l4t. - make bonsai-fallback -fi -make bonsai-grpc -make bonsai-rpc-server - -ccache -s || true diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 6d0e30e7aa21..ec5e354c5740 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -7,7 +7,6 @@ This PR fixes # **[Signed commits](../CONTRIBUTING.md#signing-off-on-commits-developer-certificate-of-origin)** - [ ] Yes, I signed my commits. -- [ ] Documentation updated (docs/content/) for user-facing changes, or not applicable