feat(gallery): let one gallery entry offer several builds of the same model

.agents/adding-backends.md

@@ -216,6 +216,69 @@ docker-build-backends: ... docker-build-<backend-name>
 - If the backend is in `backend/python/<backend-name>/` but uses `.` as context in the workflow file, use `.` context
 - Check similar backends to determine the correct context
 
+## Engine preference for gallery model variants
+
+A gallery entry can declare `variants`, alternative builds of the same weights,
+and LocalAI picks one per host: it drops builds whose backend cannot run here or
+that do not fit memory, then ranks the survivors by **engine preference
+first, serving feature second, size third** (`SelectVariant` in
+`core/gallery/resolve_variant.go`).
+
+Ask whether your backend should outrank another one on some hardware. If it
+should, add it to `engineNamePreferenceRules` in `pkg/system/capabilities.go`,
+best engine first for that capability:
+
+```go
+ 	{Nvidia, []string{engineVLLM, engineSGLang, engineLlamaCpp}},
++	{Nvidia, []string{engineVLLM, engineSGLang, engineMyEngine, engineLlamaCpp}},
+```
+
+That is the ENGINE NAME table, matched as a substring of a gallery entry's
+`backend:` value. Two sibling tables in the same file speak different
+vocabularies and are matched against different things:
+
+| Table | Vocabulary | Matched against | Consumer |
+|-------|-----------|-----------------|----------|
+| `backendBuildTagPreferenceRules` | build tags (`cuda`, `rocm`, `metal`) | installed build directory names, as a substring | alias resolution in `ListSystemBackends` |
+| `engineNamePreferenceRules` | engine names (`vllm`, `llama-cpp`, `mlx`) | a gallery entry's `backend:`, as a substring | gallery variant ranking |
+| `servingFeaturePreferenceTokens` | serving features (`dflash`, `mtp`) | a gallery entry's `tags:`, compared whole and case-insensitively, and nothing else | gallery variant ranking, one rank below the engine |
+
+**Putting a token in the wrong table matches nothing and does not error**: every
+candidate scores equal and the next sort key decides, so the preference silently
+stops existing. The block comment above all three tables spells the contract out.
+
+The serving feature table is the odd one: it is not keyed by capability, because
+no hardware prefers a plain build over an equivalent faster build of the same
+weights. It reads a declared tag and nothing else. The entry name was the
+original signal and is gone: a naming convention is not a contract, and names
+are author-supplied free text where a short marker like `mtp` turns up inside
+unrelated words or on weights whose entry enables nothing.
+`overrides.options` was rejected for the mirror-image reason: `spec_type:` is
+llama.cpp's config vocabulary, whereas a cross-backend ranking decision must
+work the same for `ds4`'s `mtp_path:` and `sglang`'s `speculative_algorithm:`.
+
+**If your backend can serve the same weights faster** (speculative decoding,
+multi-token prediction), say so in the docs for its gallery entries so curators
+tag them: the tagging rule and the per-backend evidence table live in
+[adding-gallery-models.md](adding-gallery-models.md). A backend never needs to
+appear in the token table itself; it ranks builds, not engines.
+
+Leaving your backend out is a valid choice when no ordering can be justified for
+it. It then ranks below every known engine and selection falls back to size,
+which is the behaviour that predates preference.
+
+**Leaving a whole capability out is not.** A missing row gives that host an
+empty preference list, so size alone decides among everything that survives the
+filters, and the filter will not save you: `IsBackendCompatible` derives hardware
+support from the engine NAME, so `vllm` and `sglang` carry no darwin, cuda, rocm
+or sycl token and are never dropped on a host with no GPU. That is why `default`
+(no usable accelerator, including a GPU under the 4 GiB VRAM floor) and
+`darwin-x86` both have rows putting `llama-cpp` first. Every capability
+`getSystemCapabilities()` can return needs a row unless every engine really is
+equally at home there. When you add one, enumerate the engines you are demoting
+rather than relying on them falling through unmatched: unmatched engines all tie
+with each other, so size decides among them.
+
 ## Documenting the backend (README + docs)
 
 A backend is not "added" until it is discoverable. Update the user-facing docs:
@@ -252,6 +315,7 @@ After adding a new backend, verify:
 - [ ] No Makefile syntax errors (check with linter)
 - [ ] Follows the same pattern as similar backends (e.g., if it's a transcription backend, follow `faster-whisper` pattern)
 - [ ] **`Load` validates its input and refuses models it can't serve.** When a model config has no explicit `backend:`, the model loader greedily probes *every* installed backend with the model's name and binds to the first `Load` that succeeds — an accept-anything `Load` will capture arbitrary LLMs (issue #9287). Backends that load a real artefact get this for free (the load fails); backends with no artefact must gate on the name: `opus` accepts only its own name (or none), `local-store` requires the `store.NamespacePrefix` namespace marker sent by `core/backend/stores.go`.
+- [ ] **Gallery variant ranking considered**: if this backend should be preferred over another on some hardware, it is listed in `engineNamePreferenceRules` (NOT `backendBuildTagPreferenceRules`, NOT `servingFeaturePreferenceTokens`) in `pkg/system/capabilities.go`. A missing entry silently ranks it last and lets the next sort key decide.
 - [ ] Documented: added to the category list in `docs/content/features/backends.md` (and any new endpoint/realtime capability documented under `docs/content/`)
 - [ ] If it is an in-house native C/C++/GGML engine, added to the maintained-engines table in the top-level `README.md`
 

.agents/adding-gallery-models.md

@@ -91,6 +91,106 @@ To add a variant (e.g., different quantization), use YAML merge:
       uri: huggingface://<gguf-org>/<gguf-repo>/<filename>-Q8_0.gguf
 ```
 
+## Offering several builds of one model (`variants`)
+
+When the same model is published in more than one quantization, or is also
+servable by another engine, add each build as its own ordinary gallery entry and
+then point one of them at the others with `variants`:
+
+```yaml
+- !!merge <<: *chatml
+  name: "nanbeige4.1-3b-q4"
+  # ... the usual urls / overrides / files for the Q4 build ...
+  variants:
+    - model: nanbeige4.1-3b-q8
+```
+
+Rules:
+
+- The declaring entry is a **complete, normal entry**. It keeps its own
+  `files`/`overrides` and stays installable on every host and by every older
+  LocalAI release, which simply ignore `variants`.
+- A variant references another gallery entry **by name**. That entry must exist
+  and must not declare `variants` of its own.
+- **A referenced entry keeps its own gallery row by default.** It is hidden only
+  in the collapsed listing (`collapse_variants=true`, which the web UI always
+  requests), where the declaring entry stands in for it. A search term bypasses
+  the collapse, so referencing an entry never makes it unfindable by name.
+- **Order carries no meaning.** Do not try to encode a preference; write the
+  list in whatever order reads best.
+- **A variant may be smaller than the declaring entry.** Offering a downgrade
+  for small hosts is a normal shape: the declaring entry's own build competes
+  like every other candidate, so a large host keeps the large build.
+- **Do not describe hardware.** At install time LocalAI drops variants whose
+  backend cannot run on the host, then drops those that do not fit available
+  memory. The declaring entry's own build is exempt from both filters, so
+  selection always terminates on something installable. Sizes are measured live
+  from the weights and cached, so nothing has to be written down.
+- **Engine preference outranks size.** Among the builds that survive the
+  filters, the host's preferred engine wins first and only then does the larger
+  footprint win. On NVIDIA a vLLM build beats a larger llama.cpp one; on Apple
+  silicon an MLX build beats a larger GGUF one; on a host with no preference for
+  either engine the larger build wins, since a bigger footprint is a higher
+  quality quantization of the same weights. Predict what a user gets by asking
+  which engine the host prefers before asking which build is biggest. The
+  per-capability order lives in `engineNamePreferenceRules`
+  (`pkg/system/capabilities.go`); see
+  [adding-backends.md](adding-backends.md) for how a backend gets into it.
+- **Serving feature preference sits between engine and size.** Among builds on
+  an equally preferred engine, one that speculates or predicts several tokens
+  per step beats the plain build of the same weights, because it answers faster
+  for the same output: a `dflash` build beats an `mtp` one, and either beats a
+  plain build. The order lives in `servingFeaturePreferenceTokens`
+  (`pkg/system/capabilities.go`) and is matched against the entry's `tags:` and
+  **nothing else**: not the entry name, not `overrides.options`. See
+  [the tagging rule](#the-dflash--mtp-tagging-rule) below. Engine deliberately
+  outranks it: a serving feature makes the right engine faster, it does not make
+  a wrong engine right. Fit still outranks both, so a drafter pairing (strictly
+  larger than the plain build, since it ships a drafter alongside it) is dropped
+  on a host too small for it before this order is ever consulted.
+- A variant is nothing but a name; there is no per-variant memory field. When
+  the measured size for a build is wrong, correct it on the referenced entry by
+  setting that entry's own `size:` (e.g. `size: "20GiB"`). The estimator prefers
+  a declared size over its own guesswork, so the fix applies everywhere the size
+  is shown or compared rather than only to variant selection.
+
+Users can override the automatic choice with `variant` on `POST /models/apply`,
+`local-ai models install --variant`, or the `install_model` MCP tool. See
+`docs/content/features/model-gallery.md`.
+
+The gallery lint specs live in `core/gallery`, so run that suite after adding a
+`variants` list.
+
+### The `dflash` / `mtp` tagging rule
+
+**Tag an entry `dflash` or `mtp` when the entry actually configures that
+feature. Variant ranking reads the tag and nothing else.**
+
+Decide by looking at what the entry configures, in whatever vocabulary its
+backend uses:
+
+| Backend | Configures the feature when it declares |
+|---------|------------------------------------------|
+| `llama-cpp` | `overrides.options` contains `spec_type:draft-dflash` or `spec_type:draft-mtp` |
+| `ds4` | `overrides.options` contains `mtp_path:` / `mtp_draft:` |
+| `sglang` | the referenced `gallery/*.yaml` sets `speculative_algorithm:` |
+
+That check is curation-time only. `spec_type` is llama.cpp's config vocabulary,
+and a cross-backend ranking decision must not depend on one backend's option
+syntax, which is precisely why the ranker reads the tag instead of the options.
+
+Two mistakes the rule exists to prevent:
+
+- **Weights that carry the heads are not an entry that enables them.** The
+  NVFP4 GGUF entries ship MTP-bearing weights but set only `use_jinja:true`, so
+  they enable no speculative decoding and must NOT be tagged. Tagging them wins
+  them the feature axis without being any faster.
+- **A name is not a declaration.** An entry whose name spells `-mtp` while
+  configuring nothing gets no tag, and an entry that configures the feature is
+  tagged even when its name says nothing (`hy3`, `glm-5.2`). Ranking never reads
+  the name, so an untagged build that does enable the feature is simply ranked
+  as plain rather than promoted on a marker nobody meant.
+
 ## Available template configs
 
 Look at existing `.yaml` files in `gallery/` to find the right prompt template for your model architecture:

AGENTS.md

@@ -44,4 +44,5 @@ LocalAI follows the Linux kernel project's [guidelines for AI coding assistants]
 - **Admin endpoints → MCP tool**: every admin endpoint that an admin would manage conversationally (install/list/edit/toggle/upgrade) MUST also be exposed as an MCP tool in `pkg/mcp/localaitools/`. The LocalAI Assistant chat modality and the standalone `local-ai mcp-server` consume that package; drift between REST and MCP is a real risk. Read [.agents/localai-assistant-mcp.md](.agents/localai-assistant-mcp.md) — the `TestToolHTTPRouteMappingComplete` test fails until you wire the new tool and update the route map.
 - **Build**: Inspect `Makefile` and `.github/workflows/` — ask the user before running long builds
 - **Backend OS coverage**: a new backend must target every OS it can build for, not just Linux. `.github/backend-matrix.yml` has two matrices — `include:` (Linux) and `includeDarwin:` (macOS / Apple Silicon). Most C/C++/GGML and many Python backends build on Darwin too — wire the `includeDarwin` entry + `backend/index.yaml` `metal:` entries, or say in the PR why an OS is unsupported. See the darwin checklist in [.agents/adding-backends.md](.agents/adding-backends.md).
+- **Gallery variant ranking**: a gallery entry can declare `variants` (alternative builds of the same weights), and LocalAI ranks the ones a host can run by engine preference first, size second. A new backend that should be preferred on some hardware must be listed in `engineNamePreferenceRules` in `pkg/system/capabilities.go`; the sibling `backendBuildTagPreferenceRules` speaks build tags rather than engine names, and using the wrong table matches nothing without erroring. See [.agents/adding-backends.md](.agents/adding-backends.md).
 - **UI**: The active UI is the React app in `core/http/react-ui/`. The older Alpine.js/HTML UI in `core/http/static/` is pending deprecation — all new UI work goes in the React UI

core/cli/models.go

@@ -39,6 +39,7 @@ type ModelsInstall struct {
 	RequireBackendIntegrity  bool     `env:"LOCALAI_REQUIRE_BACKEND_INTEGRITY,REQUIRE_BACKEND_INTEGRITY" help:"If true, reject backend installs without a configured signature verification policy (OCI URIs) or SHA256 (tarball/HTTP URIs)." group:"hardening" default:"false"`
 	AutoloadBackendGalleries bool     `env:"LOCALAI_AUTOLOAD_BACKEND_GALLERIES" help:"If true, automatically loads backend galleries" group:"backends" default:"true"`
 	ModelArgs                []string `arg:"" optional:"" name:"models" help:"Model configuration URLs to load"`
+	Variant                  string   `name:"variant" help:"Install a specific variant of a gallery entry that declares them, by the variant's model name. Leave unset to let LocalAI auto-select: builds this hardware cannot run or cannot fit are dropped, the engine this hardware prefers wins, and size decides among equals." group:"models"`
 
 	ModelsCMDFlags `embed:""`
 }
@@ -147,7 +148,11 @@ func (mi *ModelsInstall) Run(ctx *cliContext.Context) error {
 		}
 
 		modelLoader := model.NewModelLoader(systemState)
-		err = startup.InstallModels(context.Background(), galleryService, galleries, backendGalleries, systemState, modelLoader, !mi.DisablePredownloadScan, mi.AutoloadBackendGalleries, mi.RequireBackendIntegrity, progressCallback, modelName)
+		var installOptions []gallery.InstallOption
+		if mi.Variant != "" {
+			installOptions = append(installOptions, gallery.WithVariant(mi.Variant))
+		}
+		err = startup.InstallModelsWithOptions(context.Background(), galleryService, galleries, backendGalleries, systemState, modelLoader, !mi.DisablePredownloadScan, mi.AutoloadBackendGalleries, mi.RequireBackendIntegrity, progressCallback, installOptions, modelName)
 		if err != nil {
 			return err
 		}

core/gallery/available_memory_internal_test.go

@@ -0,0 +1,73 @@
+package gallery
+
+import (
+	"os"
+
+	"github.com/mudler/LocalAI/pkg/system"
+	"github.com/mudler/LocalAI/pkg/xsysinfo"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("availableModelMemory", func() {
+	// hostRAM is the figure the fallback resolves to. A host that cannot report
+	// its own RAM cannot exercise the fallback at all, so those specs skip
+	// rather than assert on a number that is legitimately unavailable.
+	hostRAM := func() uint64 {
+		ram, err := xsysinfo.GetSystemRAMInfo()
+		if err != nil || ram == nil || ram.Total == 0 {
+			Skip("this host does not report system RAM; the RAM fallback cannot be exercised here")
+		}
+		return ram.Total
+	}
+
+	// stateWithCapability forces DetectedCapability for one spec. The state is
+	// built fresh so nothing is served from the capability cache.
+	stateWithCapability := func(capability string, vram uint64) *system.SystemState {
+		previous, had := os.LookupEnv("LOCALAI_FORCE_META_BACKEND_CAPABILITY")
+		Expect(os.Setenv("LOCALAI_FORCE_META_BACKEND_CAPABILITY", capability)).To(Succeed())
+		DeferCleanup(func() {
+			if had {
+				Expect(os.Setenv("LOCALAI_FORCE_META_BACKEND_CAPABILITY", previous)).To(Succeed())
+				return
+			}
+			Expect(os.Unsetenv("LOCALAI_FORCE_META_BACKEND_CAPABILITY")).To(Succeed())
+		})
+
+		s := &system.SystemState{}
+		s.VRAM = vram
+		Expect(s.DetectedCapability()).To(Equal(capability), "the capability override did not take")
+		return s
+	}
+
+	// A unified-memory host reports a GPU capability but no discrete VRAM pool.
+	// Apple Silicon is the case that matters: metal is reported unconditionally
+	// on arm64 macs while TotalAvailableVRAM finds nothing to add up. Reading
+	// that zero as the budget drops every sized variant and strands the host on
+	// the base build however much memory it has, which is what turned the macOS
+	// runner red.
+	It("falls back to system RAM when a detected GPU reports no VRAM", func() {
+		ram := hostRAM()
+
+		got := availableModelMemory(stateWithCapability("metal", 0))
+
+		Expect(got).ToNot(BeZero(), "a unified-memory host was given a zero budget; every sized variant would be dropped")
+		Expect(got).To(Equal(ram))
+	})
+
+	// A discrete GPU reports real VRAM, and that stays the budget: the model
+	// lives on the card, so system RAM is not what bounds it.
+	It("prefers VRAM when the GPU reports it", func() {
+		const vram = uint64(24) << 30
+
+		Expect(availableModelMemory(stateWithCapability("nvidia-cuda-12", vram))).To(Equal(vram))
+	})
+
+	// With no GPU at all the model lives in system RAM.
+	It("uses system RAM when no GPU is detected", func() {
+		ram := hostRAM()
+
+		Expect(availableModelMemory(stateWithCapability("default", 0))).To(Equal(ram))
+	})
+})