diff --git a/.agents/adding-backends.md b/.agents/adding-backends.md index ac169b534f5d..603c9dbc80c5 100644 --- a/.agents/adding-backends.md +++ b/.agents/adding-backends.md @@ -216,6 +216,69 @@ docker-build-backends: ... docker-build- - If the backend is in `backend/python//` but uses `.` as context in the workflow file, use `.` context - Check similar backends to determine the correct context +## Engine preference for gallery model variants + +A gallery entry can declare `variants`, alternative builds of the same weights, +and LocalAI picks one per host: it drops builds whose backend cannot run here or +that do not fit memory, then ranks the survivors by **engine preference +first, serving feature second, size third** (`SelectVariant` in +`core/gallery/resolve_variant.go`). + +Ask whether your backend should outrank another one on some hardware. If it +should, add it to `engineNamePreferenceRules` in `pkg/system/capabilities.go`, +best engine first for that capability: + +```go + {Nvidia, []string{engineVLLM, engineSGLang, engineLlamaCpp}}, ++ {Nvidia, []string{engineVLLM, engineSGLang, engineMyEngine, engineLlamaCpp}}, +``` + +That is the ENGINE NAME table, matched as a substring of a gallery entry's +`backend:` value. Two sibling tables in the same file speak different +vocabularies and are matched against different things: + +| Table | Vocabulary | Matched against | Consumer | +|-------|-----------|-----------------|----------| +| `backendBuildTagPreferenceRules` | build tags (`cuda`, `rocm`, `metal`) | installed build directory names, as a substring | alias resolution in `ListSystemBackends` | +| `engineNamePreferenceRules` | engine names (`vllm`, `llama-cpp`, `mlx`) | a gallery entry's `backend:`, as a substring | gallery variant ranking | +| `servingFeaturePreferenceTokens` | serving features (`dflash`, `mtp`) | a gallery entry's `tags:`, compared whole and case-insensitively, and nothing else | gallery variant ranking, one rank below the engine | + +**Putting a token in the wrong table matches nothing and does not error**: every +candidate scores equal and the next sort key decides, so the preference silently +stops existing. The block comment above all three tables spells the contract out. + +The serving feature table is the odd one: it is not keyed by capability, because +no hardware prefers a plain build over an equivalent faster build of the same +weights. It reads a declared tag and nothing else. The entry name was the +original signal and is gone: a naming convention is not a contract, and names +are author-supplied free text where a short marker like `mtp` turns up inside +unrelated words or on weights whose entry enables nothing. +`overrides.options` was rejected for the mirror-image reason: `spec_type:` is +llama.cpp's config vocabulary, whereas a cross-backend ranking decision must +work the same for `ds4`'s `mtp_path:` and `sglang`'s `speculative_algorithm:`. + +**If your backend can serve the same weights faster** (speculative decoding, +multi-token prediction), say so in the docs for its gallery entries so curators +tag them: the tagging rule and the per-backend evidence table live in +[adding-gallery-models.md](adding-gallery-models.md). A backend never needs to +appear in the token table itself; it ranks builds, not engines. + +Leaving your backend out is a valid choice when no ordering can be justified for +it. It then ranks below every known engine and selection falls back to size, +which is the behaviour that predates preference. + +**Leaving a whole capability out is not.** A missing row gives that host an +empty preference list, so size alone decides among everything that survives the +filters, and the filter will not save you: `IsBackendCompatible` derives hardware +support from the engine NAME, so `vllm` and `sglang` carry no darwin, cuda, rocm +or sycl token and are never dropped on a host with no GPU. That is why `default` +(no usable accelerator, including a GPU under the 4 GiB VRAM floor) and +`darwin-x86` both have rows putting `llama-cpp` first. Every capability +`getSystemCapabilities()` can return needs a row unless every engine really is +equally at home there. When you add one, enumerate the engines you are demoting +rather than relying on them falling through unmatched: unmatched engines all tie +with each other, so size decides among them. + ## Documenting the backend (README + docs) A backend is not "added" until it is discoverable. Update the user-facing docs: @@ -252,6 +315,7 @@ After adding a new backend, verify: - [ ] No Makefile syntax errors (check with linter) - [ ] Follows the same pattern as similar backends (e.g., if it's a transcription backend, follow `faster-whisper` pattern) - [ ] **`Load` validates its input and refuses models it can't serve.** When a model config has no explicit `backend:`, the model loader greedily probes *every* installed backend with the model's name and binds to the first `Load` that succeeds — an accept-anything `Load` will capture arbitrary LLMs (issue #9287). Backends that load a real artefact get this for free (the load fails); backends with no artefact must gate on the name: `opus` accepts only its own name (or none), `local-store` requires the `store.NamespacePrefix` namespace marker sent by `core/backend/stores.go`. +- [ ] **Gallery variant ranking considered**: if this backend should be preferred over another on some hardware, it is listed in `engineNamePreferenceRules` (NOT `backendBuildTagPreferenceRules`, NOT `servingFeaturePreferenceTokens`) in `pkg/system/capabilities.go`. A missing entry silently ranks it last and lets the next sort key decide. - [ ] Documented: added to the category list in `docs/content/features/backends.md` (and any new endpoint/realtime capability documented under `docs/content/`) - [ ] If it is an in-house native C/C++/GGML engine, added to the maintained-engines table in the top-level `README.md` diff --git a/.agents/adding-gallery-models.md b/.agents/adding-gallery-models.md index fe5a57e87177..51221608f66a 100644 --- a/.agents/adding-gallery-models.md +++ b/.agents/adding-gallery-models.md @@ -91,6 +91,108 @@ To add a variant (e.g., different quantization), use YAML merge: uri: huggingface:////-Q8_0.gguf ``` +## Offering several builds of one model (`variants`) + +When the same model is published in more than one quantization, or is also +servable by another engine, add each build as its own ordinary gallery entry and +then point one of them at the others with `variants`: + +```yaml +- !!merge <<: *chatml + name: "nanbeige4.1-3b-q4" + # ... the usual urls / overrides / files for the Q4 build ... + variants: + - model: nanbeige4.1-3b-q8 +``` + +Rules: + +- The declaring entry is a **complete, normal entry**. It keeps its own + `files`/`overrides` and stays installable on every host and by every older + LocalAI release, which simply ignore `variants`. +- A variant references another gallery entry **by name**. That entry must exist + and must not declare `variants` of its own. +- **A referenced entry keeps its own gallery row by default.** It is hidden only + in the collapsed listing (`collapse_variants=true`, which the web UI requests + by default), where the declaring entry stands in for it. Searching there still + matches the referenced entry and answers with the entry declaring it, so + referencing an entry never makes it unfindable; turning the collapse off + returns it under its own name. +- **Order carries no meaning.** Do not try to encode a preference; write the + list in whatever order reads best. +- **A variant may be smaller than the declaring entry.** Offering a downgrade + for small hosts is a normal shape: the declaring entry's own build competes + like every other candidate, so a large host keeps the large build. +- **Do not describe hardware.** At install time LocalAI drops variants whose + backend cannot run on the host, then drops those that do not fit available + memory. The declaring entry's own build is exempt from both filters, so + selection always terminates on something installable. Sizes are measured live + from the weights and cached, so nothing has to be written down. +- **Engine preference outranks size.** Among the builds that survive the + filters, the host's preferred engine wins first and only then does the larger + footprint win. On NVIDIA a vLLM build beats a larger llama.cpp one; on Apple + silicon an MLX build beats a larger GGUF one; on a host with no preference for + either engine the larger build wins, since a bigger footprint is a higher + quality quantization of the same weights. Predict what a user gets by asking + which engine the host prefers before asking which build is biggest. The + per-capability order lives in `engineNamePreferenceRules` + (`pkg/system/capabilities.go`); see + [adding-backends.md](adding-backends.md) for how a backend gets into it. +- **Serving feature preference sits between engine and size.** Among builds on + an equally preferred engine, one that speculates or predicts several tokens + per step beats the plain build of the same weights, because it answers faster + for the same output: a `dflash` build beats an `mtp` one, and either beats a + plain build. The order lives in `servingFeaturePreferenceTokens` + (`pkg/system/capabilities.go`) and is matched against the entry's `tags:` and + **nothing else**: not the entry name, not `overrides.options`. See + [the tagging rule](#the-dflash--mtp-tagging-rule) below. Engine deliberately + outranks it: a serving feature makes the right engine faster, it does not make + a wrong engine right. Fit still outranks both, so a drafter pairing (strictly + larger than the plain build, since it ships a drafter alongside it) is dropped + on a host too small for it before this order is ever consulted. +- A variant is nothing but a name; there is no per-variant memory field. When + the measured size for a build is wrong, correct it on the referenced entry by + setting that entry's own `size:` (e.g. `size: "20GiB"`). The estimator prefers + a declared size over its own guesswork, so the fix applies everywhere the size + is shown or compared rather than only to variant selection. + +Users can override the automatic choice with `variant` on `POST /models/apply`, +`local-ai models install --variant`, or the `install_model` MCP tool. See +`docs/content/features/model-gallery.md`. + +The gallery lint specs live in `core/gallery`, so run that suite after adding a +`variants` list. + +### The `dflash` / `mtp` tagging rule + +**Tag an entry `dflash` or `mtp` when the entry actually configures that +feature. Variant ranking reads the tag and nothing else.** + +Decide by looking at what the entry configures, in whatever vocabulary its +backend uses: + +| Backend | Configures the feature when it declares | +|---------|------------------------------------------| +| `llama-cpp` | `overrides.options` contains `spec_type:draft-dflash` or `spec_type:draft-mtp` | +| `ds4` | `overrides.options` contains `mtp_path:` / `mtp_draft:` | +| `sglang` | the referenced `gallery/*.yaml` sets `speculative_algorithm:` | + +That check is curation-time only. `spec_type` is llama.cpp's config vocabulary, +and a cross-backend ranking decision must not depend on one backend's option +syntax, which is precisely why the ranker reads the tag instead of the options. + +Two mistakes the rule exists to prevent: + +- **Weights that carry the heads are not an entry that enables them.** The + NVFP4 GGUF entries ship MTP-bearing weights but set only `use_jinja:true`, so + they enable no speculative decoding and must NOT be tagged. Tagging them wins + them the feature axis without being any faster. +- **A name is not a declaration.** An entry whose name spells `-mtp` while + configuring nothing gets no tag, and an entry that configures the feature is + tagged even when its name says nothing (`hy3`, `glm-5.2`). Ranking never reads + the name, so an untagged build that does enable the feature is simply ranked + as plain rather than promoted on a marker nobody meant. + ## Available template configs Look at existing `.yaml` files in `gallery/` to find the right prompt template for your model architecture: diff --git a/AGENTS.md b/AGENTS.md index c03db98e2046..393c07845e34 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -44,4 +44,5 @@ LocalAI follows the Linux kernel project's [guidelines for AI coding assistants] - **Admin endpoints → MCP tool**: every admin endpoint that an admin would manage conversationally (install/list/edit/toggle/upgrade) MUST also be exposed as an MCP tool in `pkg/mcp/localaitools/`. The LocalAI Assistant chat modality and the standalone `local-ai mcp-server` consume that package; drift between REST and MCP is a real risk. Read [.agents/localai-assistant-mcp.md](.agents/localai-assistant-mcp.md) — the `TestToolHTTPRouteMappingComplete` test fails until you wire the new tool and update the route map. - **Build**: Inspect `Makefile` and `.github/workflows/` — ask the user before running long builds - **Backend OS coverage**: a new backend must target every OS it can build for, not just Linux. `.github/backend-matrix.yml` has two matrices — `include:` (Linux) and `includeDarwin:` (macOS / Apple Silicon). Most C/C++/GGML and many Python backends build on Darwin too — wire the `includeDarwin` entry + `backend/index.yaml` `metal:` entries, or say in the PR why an OS is unsupported. See the darwin checklist in [.agents/adding-backends.md](.agents/adding-backends.md). +- **Gallery variant ranking**: a gallery entry can declare `variants` (alternative builds of the same weights), and LocalAI ranks the ones a host can run by engine preference first, size second. A new backend that should be preferred on some hardware must be listed in `engineNamePreferenceRules` in `pkg/system/capabilities.go`; the sibling `backendBuildTagPreferenceRules` speaks build tags rather than engine names, and using the wrong table matches nothing without erroring. See [.agents/adding-backends.md](.agents/adding-backends.md). - **UI**: The active UI is the React app in `core/http/react-ui/`. The older Alpine.js/HTML UI in `core/http/static/` is pending deprecation — all new UI work goes in the React UI diff --git a/core/cli/models.go b/core/cli/models.go index a95462da1864..e2a832b0c877 100644 --- a/core/cli/models.go +++ b/core/cli/models.go @@ -39,6 +39,7 @@ type ModelsInstall struct { RequireBackendIntegrity bool `env:"LOCALAI_REQUIRE_BACKEND_INTEGRITY,REQUIRE_BACKEND_INTEGRITY" help:"If true, reject backend installs without a configured signature verification policy (OCI URIs) or SHA256 (tarball/HTTP URIs)." group:"hardening" default:"false"` AutoloadBackendGalleries bool `env:"LOCALAI_AUTOLOAD_BACKEND_GALLERIES" help:"If true, automatically loads backend galleries" group:"backends" default:"true"` ModelArgs []string `arg:"" optional:"" name:"models" help:"Model configuration URLs to load"` + Variant string `name:"variant" help:"Install a specific variant of a gallery entry that declares them, by the variant's model name. Leave unset to let LocalAI auto-select: builds this hardware cannot run or cannot fit are dropped, the engine this hardware prefers wins, and size decides among equals." group:"models"` ModelsCMDFlags `embed:""` } @@ -147,7 +148,11 @@ func (mi *ModelsInstall) Run(ctx *cliContext.Context) error { } modelLoader := model.NewModelLoader(systemState) - err = startup.InstallModels(context.Background(), galleryService, galleries, backendGalleries, systemState, modelLoader, !mi.DisablePredownloadScan, mi.AutoloadBackendGalleries, mi.RequireBackendIntegrity, progressCallback, modelName) + var installOptions []gallery.InstallOption + if mi.Variant != "" { + installOptions = append(installOptions, gallery.WithVariant(mi.Variant)) + } + err = startup.InstallModelsWithOptions(context.Background(), galleryService, galleries, backendGalleries, systemState, modelLoader, !mi.DisablePredownloadScan, mi.AutoloadBackendGalleries, mi.RequireBackendIntegrity, progressCallback, installOptions, modelName) if err != nil { return err } diff --git a/core/gallery/available_memory_internal_test.go b/core/gallery/available_memory_internal_test.go new file mode 100644 index 000000000000..e62b13800991 --- /dev/null +++ b/core/gallery/available_memory_internal_test.go @@ -0,0 +1,73 @@ +package gallery + +import ( + "os" + + "github.com/mudler/LocalAI/pkg/system" + "github.com/mudler/LocalAI/pkg/xsysinfo" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +var _ = Describe("availableModelMemory", func() { + // hostRAM is the figure the fallback resolves to. A host that cannot report + // its own RAM cannot exercise the fallback at all, so those specs skip + // rather than assert on a number that is legitimately unavailable. + hostRAM := func() uint64 { + ram, err := xsysinfo.GetSystemRAMInfo() + if err != nil || ram == nil || ram.Total == 0 { + Skip("this host does not report system RAM; the RAM fallback cannot be exercised here") + } + return ram.Total + } + + // stateWithCapability forces DetectedCapability for one spec. The state is + // built fresh so nothing is served from the capability cache. + stateWithCapability := func(capability string, vram uint64) *system.SystemState { + previous, had := os.LookupEnv("LOCALAI_FORCE_META_BACKEND_CAPABILITY") + Expect(os.Setenv("LOCALAI_FORCE_META_BACKEND_CAPABILITY", capability)).To(Succeed()) + DeferCleanup(func() { + if had { + Expect(os.Setenv("LOCALAI_FORCE_META_BACKEND_CAPABILITY", previous)).To(Succeed()) + return + } + Expect(os.Unsetenv("LOCALAI_FORCE_META_BACKEND_CAPABILITY")).To(Succeed()) + }) + + s := &system.SystemState{} + s.VRAM = vram + Expect(s.DetectedCapability()).To(Equal(capability), "the capability override did not take") + return s + } + + // A unified-memory host reports a GPU capability but no discrete VRAM pool. + // Apple Silicon is the case that matters: metal is reported unconditionally + // on arm64 macs while TotalAvailableVRAM finds nothing to add up. Reading + // that zero as the budget drops every sized variant and strands the host on + // the base build however much memory it has, which is what turned the macOS + // runner red. + It("falls back to system RAM when a detected GPU reports no VRAM", func() { + ram := hostRAM() + + got := availableModelMemory(stateWithCapability("metal", 0)) + + Expect(got).ToNot(BeZero(), "a unified-memory host was given a zero budget; every sized variant would be dropped") + Expect(got).To(Equal(ram)) + }) + + // A discrete GPU reports real VRAM, and that stays the budget: the model + // lives on the card, so system RAM is not what bounds it. + It("prefers VRAM when the GPU reports it", func() { + const vram = uint64(24) << 30 + + Expect(availableModelMemory(stateWithCapability("nvidia-cuda-12", vram))).To(Equal(vram)) + }) + + // With no GPU at all the model lives in system RAM. + It("uses system RAM when no GPU is detected", func() { + ram := hostRAM() + + Expect(availableModelMemory(stateWithCapability("default", 0))).To(Equal(ram)) + }) +}) diff --git a/core/gallery/collapse_variants.go b/core/gallery/collapse_variants.go new file mode 100644 index 000000000000..3e308b602b7f --- /dev/null +++ b/core/gallery/collapse_variants.go @@ -0,0 +1,109 @@ +package gallery + +import ( + "os" + "strings" +) + +// VariantReferencedIDs reports the IDs of entries that another entry already +// offers as one of its variants, and which therefore need no row of their own +// in a deduplicated listing: they are reachable by installing the entry that +// references them. +// +// The returned keys are GalleryModel.ID() values, so an entry is identified by +// gallery and name rather than by name alone. Two galleries may legitimately +// ship an entry of the same name, and only the one actually referenced should +// disappear. +// +// This is the key set of VariantParents; see there for the resolution rules. +func VariantReferencedIDs(models []*GalleryModel) map[string]struct{} { + parents := VariantParents(models) + referenced := make(map[string]struct{}, len(parents)) + for id := range parents { + referenced[id] = struct{}{} + } + return referenced +} + +// VariantParents maps the ID of every entry another entry already offers as one +// of its variants to the entry that offers it. It is what a collapsed listing +// needs in order to answer a match on a hidden build with the row the user can +// actually act on: the parent. +// +// An entry that declares variants of its own is never reported, even when +// something else references it. That guard is what keeps a chain from hiding a +// row the user cannot otherwise reach: parents are always visible, so every +// hidden entry is guaranteed to have a visible entry that offers it, and no +// parent this returns is itself a key. Such a reference is an authoring error +// anyway, and variant resolution already refuses to install it, but the listing +// must stay coherent in the presence of a gallery that has one rather than +// silently swallowing entries. +// +// When two entries reference the same build, the first in gallery order wins, +// so the listing is deterministic for a gallery the linter would reject. +// +// This is a pure pass over metadata already in memory. It resolves nothing over +// the network and probes no weight files: whether an entry is referenced is +// answerable from the declared names alone. +func VariantParents(models []*GalleryModel) map[string]*GalleryModel { + // FindGalleryElement resolves a reference by scanning every entry, which + // would make this quadratic over the whole gallery. The index below is the + // same lookup precomputed once, so the matching rules have to agree with + // it: case-insensitive, path separators folded to "__", and a reference + // carrying an "@" addressed against "gallery@name" instead of the bare name. + byName := make(map[string]*GalleryModel, len(models)) + byQualified := make(map[string]*GalleryModel, len(models)) + for _, m := range models { + if m == nil { + continue + } + // First match wins, mirroring FindGalleryElement's scan order, so a + // name colliding across galleries resolves the same way here as it + // does at install time. + name := variantLookupKey(m.Name) + if _, seen := byName[name]; !seen { + byName[name] = m + } + qualified := variantLookupKey(m.ID()) + if _, seen := byQualified[qualified]; !seen { + byQualified[qualified] = m + } + } + + referenced := map[string]*GalleryModel{} + for _, m := range models { + if m == nil { + continue + } + for _, v := range m.Variants { + key := variantLookupKey(v.Model) + target, ok := byQualified[key] + if !strings.Contains(key, "@") { + target, ok = byName[key] + } + // A dangling reference names nothing to hide. Reporting it is the + // linter's job, not the listing's. + if !ok || target == nil { + continue + } + if target.HasVariants() { + continue + } + // An entry naming itself would otherwise erase itself from the + // gallery entirely. + if target.ID() == m.ID() { + continue + } + if _, claimed := referenced[target.ID()]; claimed { + continue + } + referenced[target.ID()] = m + } + } + + return referenced +} + +func variantLookupKey(name string) string { + return strings.ToLower(strings.ReplaceAll(name, string(os.PathSeparator), "__")) +} diff --git a/core/gallery/collapse_variants_test.go b/core/gallery/collapse_variants_test.go new file mode 100644 index 000000000000..7c5f63eac57f --- /dev/null +++ b/core/gallery/collapse_variants_test.go @@ -0,0 +1,157 @@ +package gallery_test + +import ( + "github.com/mudler/LocalAI/core/config" + . "github.com/mudler/LocalAI/core/gallery" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +// VariantReferencedIDs answers the only question the deduplicated listing asks: +// which entries does another entry already offer, and so need no row of their +// own. Everything else is shown. +var _ = Describe("VariantReferencedIDs", func() { + entry := func(gal, name string, variants ...string) *GalleryModel { + m := &GalleryModel{} + m.Name = name + m.Gallery = config.Gallery{Name: gal} + for _, v := range variants { + m.Variants = append(m.Variants, Variant{Model: v}) + } + return m + } + + It("reports the builds a parent references and nothing else", func() { + parent := entry("test", "parent", "build-a", "build-b") + models := []*GalleryModel{ + parent, + entry("test", "build-a"), + entry("test", "build-b"), + entry("test", "standalone"), + } + + Expect(VariantReferencedIDs(models)).To(HaveLen(2)) + Expect(VariantReferencedIDs(models)).To(HaveKey("test@build-a")) + Expect(VariantReferencedIDs(models)).To(HaveKey("test@build-b")) + // The parent is installable in its own right, and a standalone entry + // is nobody's variant. + Expect(VariantReferencedIDs(models)).NotTo(HaveKey("test@parent")) + Expect(VariantReferencedIDs(models)).NotTo(HaveKey("test@standalone")) + }) + + It("never hides an entry that declares variants of its own", func() { + // A parent referencing another parent is an authoring error that + // variant resolution refuses to install. The listing still has to stay + // coherent: if the chain hid the middle entry, the build it offers + // would be unreachable, since nothing visible would mention it. + top := entry("test", "top", "middle") + middle := entry("test", "middle", "leaf") + models := []*GalleryModel{top, middle, entry("test", "leaf")} + + referenced := VariantReferencedIDs(models) + Expect(referenced).NotTo(HaveKey("test@middle"), + "hiding a parent would strand the builds only it offers") + Expect(referenced).To(HaveKey("test@leaf")) + }) + + It("ignores an entry that references itself", func() { + // Self-reference would otherwise erase the entry from the gallery. + models := []*GalleryModel{entry("test", "loop", "loop", "build")} + models = append(models, entry("test", "build")) + + Expect(VariantReferencedIDs(models)).NotTo(HaveKey("test@loop")) + Expect(VariantReferencedIDs(models)).To(HaveKey("test@build")) + }) + + It("ignores a reference that names no existing entry", func() { + // A dangling reference hides nothing; reporting it is the linter's job. + models := []*GalleryModel{entry("test", "parent", "does-not-exist")} + Expect(VariantReferencedIDs(models)).To(BeEmpty()) + }) + + It("resolves references the way install-time lookup does", func() { + // Matching has to agree with FindGalleryElement, otherwise a reference + // that installs fine would fail to hide its target. Case-insensitive, + // and an "@" addresses gallery and name together. + models := []*GalleryModel{ + entry("test", "parent", "BUILD-A", "other@build-b"), + entry("test", "build-a"), + entry("other", "build-b"), + entry("test", "build-b"), + } + + referenced := VariantReferencedIDs(models) + Expect(referenced).To(HaveKey("test@build-a")) + Expect(referenced).To(HaveKey("other@build-b")) + // The qualified reference names the other gallery's entry, so the + // same-named local entry keeps its row. + Expect(referenced).NotTo(HaveKey("test@build-b")) + }) + + It("returns an empty set for a gallery declaring no variants at all", func() { + models := []*GalleryModel{entry("test", "a"), entry("test", "b")} + Expect(VariantReferencedIDs(models)).To(BeEmpty()) + }) + + // The collapsed listing needs more than "is this hidden": to report a match + // on a hidden build it has to name the row the user can act on. + Context("VariantParents", func() { + It("names the entry that offers each hidden build", func() { + parent := entry("test", "parent", "build-a", "build-b") + models := []*GalleryModel{ + parent, + entry("test", "build-a"), + entry("test", "build-b"), + entry("test", "standalone"), + } + + parents := VariantParents(models) + Expect(parents).To(HaveLen(2)) + Expect(parents["test@build-a"]).To(BeIdenticalTo(parent)) + Expect(parents["test@build-b"]).To(BeIdenticalTo(parent)) + Expect(parents).NotTo(HaveKey("test@standalone")) + }) + + It("never names a parent that is itself hidden", func() { + // What makes substitution safe in one hop: whatever a hidden build + // resolves to is guaranteed to be a row the listing still shows, so + // no chain can substitute a user into an entry that is not there. + models := []*GalleryModel{ + entry("test", "top", "middle"), + entry("test", "middle", "leaf"), + entry("test", "leaf"), + } + + parents := VariantParents(models) + for hidden, parent := range parents { + Expect(parents).NotTo(HaveKey(parent.ID()), + "the parent of "+hidden+" is itself hidden, so substituting lands on a row nobody can see") + } + }) + + It("gives a build claimed twice a single parent, the first in order", func() { + // A gallery the linter rejects, but the listing must still be + // deterministic rather than ordered by map iteration. + first := entry("test", "first", "shared") + second := entry("test", "second", "shared") + models := []*GalleryModel{first, second, entry("test", "shared")} + + Expect(VariantParents(models)["test@shared"]).To(BeIdenticalTo(first)) + }) + + It("agrees with VariantReferencedIDs on which entries are hidden", func() { + models := []*GalleryModel{ + entry("test", "parent", "build-a"), + entry("test", "build-a"), + entry("test", "standalone"), + } + + parents := VariantParents(models) + referenced := VariantReferencedIDs(models) + Expect(parents).To(HaveLen(len(referenced))) + for id := range referenced { + Expect(parents).To(HaveKey(id)) + } + }) + }) +}) diff --git a/core/gallery/describe_variants.go b/core/gallery/describe_variants.go new file mode 100644 index 000000000000..3b579d7df44e --- /dev/null +++ b/core/gallery/describe_variants.go @@ -0,0 +1,120 @@ +package gallery + +// VariantView is one selectable build of an entry, as a client sees it. +// +// It is the read-only mirror of what selection decides at install time, so a +// picker can show the user the same facts the installer acts on rather than a +// second, independently-derived opinion that could disagree with it. +type VariantView struct { + // Model is the name to send back as the install request's `variant`. + Model string `json:"model"` + // Backend is the engine the referenced entry resolves to. A client renders + // it, and it is also the reason Fits may be false on a host whose memory + // would be ample. + Backend string `json:"backend"` + // MemoryBytes is the measured footprint. It is omitted from the JSON + // entirely when the size could not be determined, rather than serialized as + // a zero that a client would have to know to read as unknown. An absent key + // never means "needs nothing". + MemoryBytes uint64 `json:"memory_bytes,omitempty"` + // Fits reports whether auto-selection would consider this variant on this + // host: its backend can run here and its known footprint is within budget. + // An unknown footprint counts as fitting, exactly as selection treats it. + Fits bool `json:"fits"` + // IsBase marks the entry's own payload. It is always installable and is + // what auto-selection falls back to, which is why it is listed alongside + // the declared variants rather than hidden. + IsBase bool `json:"is_base"` + // Quantization is the weight format the referenced entry installs, e.g. + // "Q2_G64", "PQ2_0", "F16". It is the fact that actually separates two + // builds of one model: name, backend and probed size routinely agree + // between variants that differ entirely in precision. + // + // Derived server-side from the referenced entry's model filename rather + // than parsed by clients, so every client reads the same format out of the + // same file the installer will point the backend at, and a naming + // convention change is one edit here rather than one per client. + // + // Omitted when the entry names no recognisable format. That is a normal + // outcome for a backend served from a directory of weights, and clients + // must render its absence rather than an empty or invented value. + Quantization string `json:"quantization,omitempty"` + // Features are the serving features this build declares, best first, e.g. + // ["dflash"]. They mean the same weights are served faster, which is a + // reason to prefer a variant that neither its size nor its precision + // conveys. + // + // This is the SAME tag-against-vocabulary match servingFeatureRank ranks + // on, over the same host preference list, so what a client shows as a speed + // advantage cannot disagree with what selection actually rewarded. A tag + // outside that vocabulary is not a serving feature and is not reported + // here: the entry's own tag list already carries it. + Features []string `json:"features,omitempty"` +} + +// EntryVariants is the variant surface of a single gallery entry. +type EntryVariants struct { + Variants []VariantView `json:"variants"` + // AutoSelected is the variant auto-selection would install on this host + // right now. Clients show it as the default choice. + AutoSelected string `json:"auto_selected"` +} + +// DescribeVariants reports an entry's selectable builds and which one +// auto-selection would currently pick. +// +// It runs the SAME variantOptions + SelectVariant pass the installer runs, so +// the reported auto-selection cannot drift from what installing would actually +// do. The env carries the same probe seam too, so the size shown here and the +// size the installer compares against come from one cache and one round trip. +// +// An entry that declares no variants returns nil, nil WITHOUT touching the +// probe. That is load-bearing: the gallery listing walks entries by the +// thousand and the overwhelming majority declare nothing, so the no-variant +// case must cost nothing at all. +func DescribeVariants(models []*GalleryModel, entry *GalleryModel, env ResolveEnv) (*EntryVariants, error) { + if entry == nil || !entry.HasVariants() { + return nil, nil + } + + options, err := variantOptions(models, entry, env) + if err != nil { + return nil, err + } + + selection, err := SelectVariant(options, env, "") + if err != nil { + return nil, err + } + + views := make([]VariantView, 0, len(options)) + for _, o := range options { + memory, known := o.EffectiveMemory() + view := VariantView{ + Model: o.Variant.Model, + Backend: o.Backend, + // The base is exempt from every filter and always installs, so + // reporting it as anything but fitting would misdescribe it. + Fits: o.IsBase || (env.backendRuns(o.Backend) && (!known || memory <= env.AvailableMemory)), + IsBase: o.IsBase, + // o.Tags is already the REFERENCED entry's tag list, which is the + // only correct source: the feature belongs to the build that would + // be installed, not to the family the parent describes. + Features: env.servingFeaturesOf(o), + } + if known { + view.MemoryBytes = memory + } + // The base option's payload is the entry itself; every other option + // names a gallery entry variantOptions has already proved resolvable, + // so this second lookup is an in-memory map hit and cannot fail. + source := entry + if !o.IsBase { + source = FindGalleryElement(models, o.Variant.Model) + } + view.Quantization = quantizationOfEntry(source) + views = append(views, view) + } + + return &EntryVariants{Variants: views, AutoSelected: selection.Option.Variant.Model}, nil +} diff --git a/core/gallery/describe_variants_test.go b/core/gallery/describe_variants_test.go new file mode 100644 index 000000000000..fb6ead9dc77a --- /dev/null +++ b/core/gallery/describe_variants_test.go @@ -0,0 +1,369 @@ +package gallery_test + +import ( + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/mudler/LocalAI/core/gallery" +) + +var _ = Describe("DescribeVariants", func() { + gib := func(n uint64) uint64 { return n * 1024 * 1024 * 1024 } + + newModel := func(name, backend string) *gallery.GalleryModel { + m := &gallery.GalleryModel{} + m.Name = name + m.Backend = backend + return m + } + + var models []*gallery.GalleryModel + var base *gallery.GalleryModel + // probed records every entry the probe was asked about, so a spec can + // assert on the ABSENCE of a round trip and not merely on the result. + var probed []string + + // probing builds an env whose sizes are injected rather than measured, so + // these specs pin exact footprints without reaching the network. + probing := func(available uint64, sizes map[string]uint64) gallery.ResolveEnv { + return gallery.ResolveEnv{ + AvailableMemory: available, + BackendCompatible: func(string) bool { return true }, + ProbeMemory: func(target *gallery.GalleryModel) uint64 { + probed = append(probed, target.Name) + return sizes[target.Name] + }, + } + } + + byName := func(view *gallery.EntryVariants, name string) gallery.VariantView { + GinkgoHelper() + for _, v := range view.Variants { + if v.Model == name { + return v + } + } + Fail("no variant named " + name + " in the reported view") + return gallery.VariantView{} + } + + BeforeEach(func() { + probed = nil + big := newModel("qwen3-8b-vllm-awq", "vllm") + small := newModel("qwen3-8b-gguf-q8", "llama-cpp") + base = newModel("qwen3-8b-gguf-q4", "llama-cpp") + base.Variants = []gallery.Variant{ + {Model: "qwen3-8b-vllm-awq"}, + {Model: "qwen3-8b-gguf-q8"}, + } + models = []*gallery.GalleryModel{big, small, base} + }) + + Describe("an entry that declares no variants", func() { + It("reports nothing and issues no probe at all", func() { + // This is the performance contract of the gallery listing: the + // listing walks over a thousand entries and virtually none of them + // declare variants, so the ordinary entry must cost zero round + // trips. Asserting on `probed` rather than on timing makes that + // contract testable. + plain := newModel("plain-entry", "llama-cpp") + models = append(models, plain) + + view, err := gallery.DescribeVariants(models, plain, probing(gib(24), map[string]uint64{ + "plain-entry": gib(4), + })) + + Expect(err).ToNot(HaveOccurred()) + Expect(view).To(BeNil()) + Expect(probed).To(BeEmpty()) + }) + + It("reports nothing for a nil entry rather than panicking", func() { + view, err := gallery.DescribeVariants(models, nil, probing(gib(24), nil)) + Expect(err).ToNot(HaveOccurred()) + Expect(view).To(BeNil()) + Expect(probed).To(BeEmpty()) + }) + }) + + Describe("an entry that declares variants", func() { + It("reports every declared variant plus the entry's own build", func() { + view, err := gallery.DescribeVariants(models, base, probing(gib(24), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(20), + "qwen3-8b-gguf-q8": gib(9), + })) + Expect(err).ToNot(HaveOccurred()) + + names := []string{} + for _, v := range view.Variants { + names = append(names, v.Model) + } + // The base is listed so a picker can offer "decline the upgrade", + // which is a real choice an operator makes. + Expect(names).To(ConsistOf("qwen3-8b-vllm-awq", "qwen3-8b-gguf-q8", "qwen3-8b-gguf-q4")) + Expect(byName(view, "qwen3-8b-gguf-q4").IsBase).To(BeTrue()) + Expect(byName(view, "qwen3-8b-vllm-awq").IsBase).To(BeFalse()) + }) + + It("reports the backend of the referenced entry, not of the declaring one", func() { + // A picker renders this, and it is also the reason a variant can be + // unavailable on a host with ample memory. + view, err := gallery.DescribeVariants(models, base, probing(gib(24), nil)) + Expect(err).ToNot(HaveOccurred()) + Expect(byName(view, "qwen3-8b-vllm-awq").Backend).To(Equal("vllm")) + Expect(byName(view, "qwen3-8b-gguf-q8").Backend).To(Equal("llama-cpp")) + }) + + It("reports the probed size of each variant", func() { + view, err := gallery.DescribeVariants(models, base, probing(gib(24), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(20), + "qwen3-8b-gguf-q8": gib(9), + })) + Expect(err).ToNot(HaveOccurred()) + Expect(byName(view, "qwen3-8b-vllm-awq").MemoryBytes).To(Equal(gib(20))) + Expect(byName(view, "qwen3-8b-gguf-q8").MemoryBytes).To(Equal(gib(9))) + }) + + It("reports a size it could not determine as unknown rather than as free", func() { + // Zero on the wire means unknown. Rendering it as a zero-byte model + // would advertise the largest download on offer as costless. + view, err := gallery.DescribeVariants(models, base, probing(gib(24), map[string]uint64{})) + Expect(err).ToNot(HaveOccurred()) + Expect(byName(view, "qwen3-8b-vllm-awq").MemoryBytes).To(Equal(uint64(0))) + Expect(byName(view, "qwen3-8b-vllm-awq").Fits).To(BeTrue()) + }) + + It("marks a variant too large for this host as not fitting", func() { + view, err := gallery.DescribeVariants(models, base, probing(gib(10), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(20), + "qwen3-8b-gguf-q8": gib(9), + })) + Expect(err).ToNot(HaveOccurred()) + Expect(byName(view, "qwen3-8b-vllm-awq").Fits).To(BeFalse()) + Expect(byName(view, "qwen3-8b-gguf-q8").Fits).To(BeTrue()) + }) + + It("marks a variant whose backend cannot run here as not fitting, however much memory there is", func() { + env := probing(gib(1024), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(20), + "qwen3-8b-gguf-q8": gib(9), + }) + env.BackendCompatible = func(backend string) bool { return backend != "vllm" } + + view, err := gallery.DescribeVariants(models, base, env) + Expect(err).ToNot(HaveOccurred()) + Expect(byName(view, "qwen3-8b-vllm-awq").Fits).To(BeFalse()) + Expect(byName(view, "qwen3-8b-gguf-q8").Fits).To(BeTrue()) + }) + + It("always reports the entry's own build as fitting", func() { + // The base is exempt from every filter and always installs, so + // showing it as unavailable would misdescribe it. Neither a hostile + // backend gate nor a zero memory budget may change that. + env := probing(0, map[string]uint64{}) + env.BackendCompatible = func(string) bool { return false } + + view, err := gallery.DescribeVariants(models, base, env) + Expect(err).ToNot(HaveOccurred()) + Expect(byName(view, "qwen3-8b-gguf-q4").Fits).To(BeTrue()) + }) + + It("surfaces a variant that references an entry no gallery declares", func() { + base.Variants = []gallery.Variant{{Model: "does-not-exist"}} + _, err := gallery.DescribeVariants(models, base, probing(gib(24), nil)) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("does-not-exist")) + }) + }) + + Describe("the reported auto-selection", func() { + // These are the specs that keep a picker honest: what the listing shows + // as the default must be what installing with no variant actually does. + // They assert against ResolveVariant rather than against a hardcoded + // name, so a change to the selection rules cannot make the two drift + // without failing here. + agreesWithInstall := func(env gallery.ResolveEnv) { + GinkgoHelper() + view, err := gallery.DescribeVariants(models, base, env) + Expect(err).ToNot(HaveOccurred()) + + _, chosen, err := gallery.ResolveVariant(models, base, env, "") + Expect(err).ToNot(HaveOccurred()) + Expect(view.AutoSelected).To(Equal(chosen.Model)) + } + + It("matches what installing would pick when everything fits", func() { + agreesWithInstall(probing(gib(64), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(20), + "qwen3-8b-gguf-q8": gib(9), + })) + }) + + It("matches what installing would pick when only the smaller variant fits", func() { + agreesWithInstall(probing(gib(12), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(20), + "qwen3-8b-gguf-q8": gib(9), + })) + }) + + It("matches what installing would pick when nothing fits and the base wins", func() { + agreesWithInstall(probing(gib(2), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(20), + "qwen3-8b-gguf-q8": gib(9), + })) + }) + + It("names the largest fitting variant, not the first authored", func() { + // Pinned separately from agreesWithInstall so that a mutation making + // BOTH functions pick the wrong variant still fails a spec. + view, err := gallery.DescribeVariants(models, base, probing(gib(64), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(20), + "qwen3-8b-gguf-q8": gib(9), + })) + Expect(err).ToNot(HaveOccurred()) + Expect(view.AutoSelected).To(Equal("qwen3-8b-vllm-awq")) + }) + + It("matches what installing would pick when the host prefers an engine", func() { + // The picker and the installer both have to apply engine + // preference, or a Mac would be shown the GGUF build and handed the + // MLX one. Asserting the agreement AND the name pins both halves. + mlx := newModel("qwen3-8b-mlx-4bit", "mlx") + models = append(models, mlx) + base.Variants = append(base.Variants, gallery.Variant{Model: "qwen3-8b-mlx-4bit"}) + + env := probing(gib(64), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(20), + "qwen3-8b-gguf-q8": gib(9), + "qwen3-8b-mlx-4bit": gib(5), + }) + // Engine names as SystemState.EnginePreferenceTokens reports them for + // metal. Build tags would match no gallery `backend:` value here. + env.EnginePreference = []string{"mlx", "llama-cpp"} + + agreesWithInstall(env) + + view, err := gallery.DescribeVariants(models, base, env) + Expect(err).ToNot(HaveOccurred()) + Expect(view.AutoSelected).To(Equal("qwen3-8b-mlx-4bit")) + }) + + It("matches what installing would pick when the host prefers vLLM", func() { + // The NVIDIA rule the user asked for, checked through the listing so + // the picker and the installer cannot drift on it. The GGUF build is + // deliberately the LARGER one, so only preference can produce this + // answer: size alone would name the llama.cpp build. + env := probing(gib(64), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(9), + "qwen3-8b-gguf-q8": gib(20), + }) + env.EnginePreference = []string{"vllm", "sglang", "llama-cpp"} + + agreesWithInstall(env) + + view, err := gallery.DescribeVariants(models, base, env) + Expect(err).ToNot(HaveOccurred()) + Expect(view.AutoSelected).To(Equal("qwen3-8b-vllm-awq")) + }) + + It("names the entry itself when no variant fits", func() { + view, err := gallery.DescribeVariants(models, base, probing(gib(2), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(20), + "qwen3-8b-gguf-q8": gib(9), + })) + Expect(err).ToNot(HaveOccurred()) + Expect(view.AutoSelected).To(Equal("qwen3-8b-gguf-q4")) + }) + }) + + Describe("the facts that tell two builds of one model apart", func() { + // servedAs points an entry at a weight file the way the gallery does, + // so the reported quantization comes from the same field the installer + // hands the backend. + servedAs := func(m *gallery.GalleryModel, filename string) { + m.Overrides = map[string]any{ + "parameters": map[string]any{"model": filename}, + } + } + + It("reports each variant's quantization from the entry it references", func() { + // The whole point of the field: these three rows share a backend + // and sit within a gigabyte of each other, so quantization is the + // only thing a user can actually choose on. + servedAs(models[0], "models/Qwen3-8B-AWQ-4bit.safetensors") + models[0].Backend = "llama-cpp" + servedAs(models[1], "models/Qwen3-8B-Q8_0.gguf") + servedAs(base, "models/Qwen3-8B-Q4_K_M.gguf") + + view, err := gallery.DescribeVariants(models, base, probing(gib(64), map[string]uint64{})) + + Expect(err).ToNot(HaveOccurred()) + Expect(byName(view, "qwen3-8b-vllm-awq").Quantization).To(Equal("4BIT")) + Expect(byName(view, "qwen3-8b-gguf-q8").Quantization).To(Equal("Q8_0")) + // The base is described from the entry's own payload, not skipped: + // it is a selectable build like any other. + Expect(byName(view, "qwen3-8b-gguf-q4").Quantization).To(Equal("Q4_K_M")) + }) + + It("leaves the quantization unset when the referenced entry names none", func() { + // A vLLM build served from a directory of safetensors declares no + // format. The field must be absent rather than empty-but-present, + // so a client renders "unknown" instead of a blank cell. + servedAs(models[0], "models/Qwen3-8B/") + servedAs(base, "models/Qwen3-8B-Q4_K_M.gguf") + + view, err := gallery.DescribeVariants(models, base, probing(gib(64), map[string]uint64{})) + + Expect(err).ToNot(HaveOccurred()) + Expect(byName(view, "qwen3-8b-vllm-awq").Quantization).To(BeEmpty()) + Expect(byName(view, "qwen3-8b-gguf-q4").Quantization).To(Equal("Q4_K_M")) + }) + + It("names the serving features a build declares, best first", func() { + models[0].Tags = []string{"llm", "MTP", "gguf", "dflash"} + + env := probing(gib(64), map[string]uint64{}) + env.ServingFeaturePreference = []string{"dflash", "mtp"} + + view, err := gallery.DescribeVariants(models, base, env) + + Expect(err).ToNot(HaveOccurred()) + // Preference order, not the author's order, and case-folded the way + // the ranker folds it: "MTP" and "mtp" declare one feature. + Expect(byName(view, "qwen3-8b-vllm-awq").Features).To(Equal([]string{"dflash", "mtp"})) + // A tag outside the vocabulary is not a serving feature. + Expect(byName(view, "qwen3-8b-gguf-q8").Features).To(BeEmpty()) + }) + + It("agrees with the ranking: the build shown as faster is the one selection rewards", func() { + // The contract that keeps the badge honest. Both builds fit and are + // runnable, so only the serving feature can decide, and the variant + // carrying the reported feature must be the one auto-selected. + models[1].Tags = []string{"dflash"} + + env := probing(gib(64), map[string]uint64{ + "qwen3-8b-vllm-awq": gib(9), + "qwen3-8b-gguf-q8": gib(9), + }) + env.ServingFeaturePreference = []string{"dflash", "mtp"} + + view, err := gallery.DescribeVariants(models, base, env) + + Expect(err).ToNot(HaveOccurred()) + Expect(byName(view, "qwen3-8b-gguf-q8").Features).To(Equal([]string{"dflash"})) + Expect(view.AutoSelected).To(Equal("qwen3-8b-gguf-q8")) + }) + + It("reports no features on a host with no serving feature vocabulary", func() { + // An env with no preference list ranks every build equally on this + // axis, so claiming a speed advantage would describe an advantage + // nothing acted on. + models[0].Tags = []string{"dflash"} + + view, err := gallery.DescribeVariants(models, base, probing(gib(64), map[string]uint64{})) + + Expect(err).ToNot(HaveOccurred()) + Expect(byName(view, "qwen3-8b-vllm-awq").Features).To(BeEmpty()) + }) + }) +}) diff --git a/core/gallery/empty_base_install_test.go b/core/gallery/empty_base_install_test.go new file mode 100644 index 000000000000..6b104e233f02 --- /dev/null +++ b/core/gallery/empty_base_install_test.go @@ -0,0 +1,243 @@ +package gallery_test + +import ( + "context" + "fmt" + "os" + "path/filepath" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "gopkg.in/yaml.v3" + + "github.com/mudler/LocalAI/core/config" + "github.com/mudler/LocalAI/core/gallery" + "github.com/mudler/LocalAI/pkg/system" +) + +// An entry declaring neither url: nor config_file: is installed on an empty +// base config, with overrides: and files: supplying everything. These specs +// cover that path, the payload rule that still rejects an entry with nothing to +// install, and the two older paths, which must be untouched. +// +// Nothing here reaches the network. The whole point of the empty-base path is +// that it fetches nothing, and a spec that quietly went to GitHub would be +// asserting the opposite of the change. +var _ = Describe("InstallModelFromGallery with an empty base config", func() { + var tempdir string + var galleries []config.Gallery + var systemState *system.SystemState + // The gallery listing is cached on the name and URL pair, so every spec + // needs a gallery of its own or it reads the previous spec's catalog. + galleryRevision := 0 + + newGallery := func(entries ...gallery.GalleryModel) { + out, err := yaml.Marshal(entries) + Expect(err).ToNot(HaveOccurred()) + name := fmt.Sprintf("empty-base-%d", galleryRevision) + galleryRevision++ + galleryPath := filepath.Join(tempdir, name+".yaml") + Expect(os.WriteFile(galleryPath, out, 0600)).To(Succeed()) + galleries = []config.Gallery{{Name: name, URL: "file://" + galleryPath}} + } + + install := func(name string, req gallery.GalleryModel, options ...gallery.InstallOption) error { + return gallery.InstallModelFromGallery( + context.TODO(), galleries, []config.Gallery{}, systemState, nil, + name, req, func(string, string, string, float64) {}, false, false, false, options...) + } + + installedConfig := func(name string) map[string]any { + dat, err := os.ReadFile(filepath.Join(tempdir, name+".yaml")) + Expect(err).ToNot(HaveOccurred()) + content := map[string]any{} + Expect(yaml.Unmarshal(dat, &content)).To(Succeed()) + return content + } + + // localWeights lets a spec carry a files: list without leaving the + // filesystem. The downloader treats an already-present destination with no + // declared sha256 as fetched and skips it, so seeding the file is what keeps + // these specs off the network. The URI is still authored, because the + // installer walks the list either way and a missing one would not exercise + // the same code. + localWeights := func(name string) gallery.File { + Expect(os.WriteFile(filepath.Join(tempdir, name), []byte("weights for "+name), 0600)).To(Succeed()) + return gallery.File{Filename: name, URI: "https://example.com/" + name} + } + + BeforeEach(func() { + var err error + tempdir, err = os.MkdirTemp("", "empty-base-install") + Expect(err).ToNot(HaveOccurred()) + DeferCleanup(func() { Expect(os.RemoveAll(tempdir)).To(Succeed()) }) + + systemState, err = system.GetSystemState(system.WithModelPath(tempdir)) + Expect(err).ToNot(HaveOccurred()) + }) + + It("installs an entry that declares only overrides and files", func() { + e := gallery.GalleryModel{Overrides: map[string]any{ + "backend": "ds4", + "parameters": map[string]any{"model": "weights.gguf"}, + }} + e.Name = "overrides-only" + e.Description = "an entry with no base config" + e.AdditionalFiles = []gallery.File{localWeights("weights.gguf")} + newGallery(e) + + Expect(install("overrides-only", gallery.GalleryModel{})).To(Succeed()) + + cfg := installedConfig("overrides-only") + Expect(cfg["backend"]).To(Equal("ds4")) + Expect(cfg["parameters"]).To(HaveKeyWithValue("model", "weights.gguf")) + // The name comes from the install, never from a base config, which is + // what the several hundred virtual.yaml entries were getting wrong: they + // inherited the stub's name. + Expect(cfg["name"]).To(Equal("overrides-only")) + Expect(filepath.Join(tempdir, "weights.gguf")).To(BeAnExistingFile()) + }) + + It("installs an entry that declares only files", func() { + e := gallery.GalleryModel{} + e.Name = "files-only" + e.AdditionalFiles = []gallery.File{localWeights("plain.gguf")} + newGallery(e) + + Expect(install("files-only", gallery.GalleryModel{})).To(Succeed()) + Expect(filepath.Join(tempdir, "plain.gguf")).To(BeAnExistingFile()) + }) + + // Half the point of the change is that the empty-base path costs no fetch, + // and an assertion that nothing was fetched is worthless unless something + // could have been. So the control runs the same install with a url: pointing + // at a base config that is not there: it must fail, proving a url IS read on + // this path, and only then does the same fixture without the url passing + // mean the read was skipped rather than silently succeeding. + Describe("the base config fetch", func() { + var missing string + + BeforeEach(func() { + missing = "file://" + filepath.Join(tempdir, "no-such-base.yaml") + Expect(filepath.Join(tempdir, "no-such-base.yaml")).ToNot(BeAnExistingFile()) + }) + + It("is attempted when the entry declares a url", func() { + e := gallery.GalleryModel{Overrides: map[string]any{"backend": "ds4"}} + e.Name = "with-url" + e.URL = missing + newGallery(e) + + // If this ever starts passing, the control has stopped controlling + // and the spec below proves nothing. + Expect(install("with-url", gallery.GalleryModel{})).To(HaveOccurred()) + }) + + It("is skipped entirely when the entry declares none", func() { + e := gallery.GalleryModel{Overrides: map[string]any{"backend": "ds4"}} + e.Name = "without-url" + newGallery(e) + + Expect(install("without-url", gallery.GalleryModel{})).To(Succeed()) + Expect(installedConfig("without-url")["backend"]).To(Equal("ds4")) + }) + }) + + Describe("an entry with nothing to install", func() { + // No url, no config_file, no overrides and no files. Accepting this + // would write an empty model directory and report success, which is the + // authoring mistake the relaxation would otherwise hide. + emptyEntry := func() gallery.GalleryModel { + e := gallery.GalleryModel{} + e.Name = "hollow" + // urls: is the informational link list. It reads like a payload and + // is not one. + e.URLs = []string{"https://huggingface.co/example/hollow"} + return e + } + + It("is refused rather than installed empty", func() { + newGallery(emptyEntry()) + + err := install("hollow", gallery.GalleryModel{}) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("installs nothing")) + Expect(err.Error()).To(ContainSubstring("hollow")) + Expect(filepath.Join(tempdir, "hollow.yaml")).ToNot(BeAnExistingFile()) + }) + + It("is accepted when the caller's request supplies the payload", func() { + // The request's overrides are merged into the install exactly as the + // entry's own are, so a caller who brings them really has asked for + // something installable. + newGallery(emptyEntry()) + + req := gallery.GalleryModel{Overrides: map[string]any{"backend": "llama-cpp"}} + Expect(install("hollow", req)).To(Succeed()) + Expect(installedConfig("hollow")["backend"]).To(Equal("llama-cpp")) + }) + }) + + // The two older paths are meant to be untouched by the relaxation. + Describe("the pre-existing paths", func() { + It("still installs an entry described by an inline config_file", func() { + e := gallery.GalleryModel{ConfigFile: map[string]any{"backend": "llama-cpp"}} + e.Name = "inline" + newGallery(e) + + Expect(install("inline", gallery.GalleryModel{})).To(Succeed()) + Expect(installedConfig("inline")["backend"]).To(Equal("llama-cpp")) + }) + + It("still installs an entry described by a url", func() { + payload, err := yaml.Marshal(gallery.ModelConfig{ + Name: "fetched", + ConfigFile: "backend: vllm\n", + }) + Expect(err).ToNot(HaveOccurred()) + payloadPath := filepath.Join(tempdir, "payload.yaml") + Expect(os.WriteFile(payloadPath, payload, 0600)).To(Succeed()) + + e := gallery.GalleryModel{} + e.Name = "from-url" + e.URL = "file://" + payloadPath + newGallery(e) + + Expect(install("from-url", gallery.GalleryModel{})).To(Succeed()) + Expect(installedConfig("from-url")["backend"]).To(Equal("vllm")) + }) + }) + + // One of the entries that shipped uninstallable, driven through the real + // install path rather than only checked as text. Its files: are swapped for + // a local one because the real ones are gigabytes on HuggingFace; its + // overrides: are the catalog's own, so this proves the authored payload + // lands. + It("installs a previously broken index entry once the empty base is allowed", func() { + entries, err := loadGalleryIndex() + Expect(err).ToNot(HaveOccurred()) + + var e gallery.GalleryModel + for _, candidate := range entries { + if candidate.Name == "liquidai_lfm2-1.2b-rag" { + e = candidate + break + } + } + Expect(e.Name).To(Equal("liquidai_lfm2-1.2b-rag")) + // The defect: no base config of any kind, which used to be fatal. + Expect(e.URL).To(BeEmpty()) + Expect(e.ConfigFile).To(BeEmpty()) + Expect(e.Overrides).ToNot(BeEmpty()) + + e.AdditionalFiles = []gallery.File{localWeights("LiquidAI_LFM2-1.2B-RAG-Q4_K_M.gguf")} + newGallery(e) + + Expect(install(e.Name, gallery.GalleryModel{})).To(Succeed()) + cfg := installedConfig(e.Name) + Expect(cfg["name"]).To(Equal(e.Name)) + // The catalog's own overrides, verbatim, laid over the empty base. + Expect(cfg["parameters"]).To(Equal(e.Overrides["parameters"])) + Expect(cfg["known_usecases"]).To(Equal(e.Overrides["known_usecases"])) + }) +}) diff --git a/core/gallery/gallery.go b/core/gallery/gallery.go index 52ca969c8294..fefd7dc4bf35 100644 --- a/core/gallery/gallery.go +++ b/core/gallery/gallery.go @@ -334,6 +334,27 @@ var ( // invalidate entries when the gallery data changes. func GalleryGeneration() uint64 { return galleryGeneration.Load() } +// ResetGalleryModelCache drops the cached model list, once any background +// refresh already in flight has finished writing to it. +// +// It exists for tests. The cache is a package global keyed by nothing, which is +// right for a process serving one gallery configuration and wrong for a suite +// where each spec stands up its own: a refresh one spec triggered can land in +// the middle of the next and answer it with the previous spec's entries, so +// whichever assertion happens to straddle it fails at random. +// +// Waiting for the in-flight refresh rather than only clearing is the point. The +// refresh publishes its result after this call would otherwise have returned, +// so clearing without waiting just narrows the window. +func ResetGalleryModelCache() { + for refreshing.Load() { + time.Sleep(time.Millisecond) + } + availableModelsMu.Lock() + availableModelsCache = nil + availableModelsMu.Unlock() +} + // AvailableGalleryModelsCached returns gallery models from an in-memory cache. // Local-only fields (installed status) are refreshed on every call. A background // goroutine is triggered to re-fetch the full model list (including network diff --git a/core/gallery/model_artifacts.go b/core/gallery/model_artifacts.go index 07924d9dfc16..b68cbab17f4b 100644 --- a/core/gallery/model_artifacts.go +++ b/core/gallery/model_artifacts.go @@ -19,6 +19,7 @@ type ArtifactMaterializer interface { type installOptions struct { materializer ArtifactMaterializer + variant string } type InstallOption func(*installOptions) @@ -31,6 +32,16 @@ func WithArtifactMaterializer(materializer ArtifactMaterializer) InstallOption { } } +// WithVariant pins a gallery entry to a specific variant by name, bypassing +// hardware-based selection. The entry's own name is a valid pin, since the +// entry is itself the last resort. Ignored for entries that declare no +// variants. +func WithVariant(variant string) InstallOption { + return func(options *installOptions) { + options.variant = variant + } +} + func applyInstallOptions(options ...InstallOption) installOptions { result := installOptions{materializer: modelartifacts.NewDefaultManager()} for _, option := range options { diff --git a/core/gallery/models.go b/core/gallery/models.go index 4d39cc0cc7a4..f3184648b15e 100644 --- a/core/gallery/models.go +++ b/core/gallery/models.go @@ -9,6 +9,7 @@ import ( "path/filepath" "slices" "strings" + "time" "dario.cat/mergo" lconfig "github.com/mudler/LocalAI/core/config" @@ -17,6 +18,8 @@ import ( "github.com/mudler/LocalAI/pkg/modelartifacts" "github.com/mudler/LocalAI/pkg/system" "github.com/mudler/LocalAI/pkg/utils" + "github.com/mudler/LocalAI/pkg/vram" + "github.com/mudler/LocalAI/pkg/xsysinfo" "github.com/mudler/xlog" "gopkg.in/yaml.v3" @@ -60,6 +63,13 @@ type ModelConfig struct { ConfigFile string `yaml:"config_file"` Files []File `yaml:"files"` PromptTemplates []PromptTemplate `yaml:"prompt_templates"` + + // The fields below record how an entry carrying variants was resolved, + // so a reinstall or upgrade can honor the same pin and so operators can + // see which variant a stable model name is actually backed by. + EntryName string `yaml:"entry_name,omitempty"` + ResolvedVariant string `yaml:"resolved_variant,omitempty"` + PinnedVariant string `yaml:"pinned_variant,omitempty"` } type File struct { @@ -73,6 +83,292 @@ type PromptTemplate struct { Content string `yaml:"content"` } +// variantOptions turns an entry's declared variants into selectable options, +// resolving each referenced gallery entry so the selector gets the backend it +// filters on, and appends the entry itself as the base option. +// +// The base is appended rather than special-cased downstream so there is exactly +// one selection path, and so an operator can pin the entry's own name to +// decline an upgrade their hardware would otherwise take. +func variantOptions(models []*GalleryModel, entry *GalleryModel, env ResolveEnv) ([]VariantOption, error) { + options := make([]VariantOption, 0, len(entry.Variants)+1) + for _, v := range entry.Variants { + // Every referenced entry is looked up here rather than lazily after + // selection, because the backend that decides whether a variant is even + // eligible lives on that entry, not on the variant. + target := FindGalleryElement(models, v.Model) + if target == nil { + return nil, fmt.Errorf("model %q references variant %q which does not exist in any configured gallery", entry.Name, v.Model) + } + if target.HasVariants() { + return nil, fmt.Errorf("model %q references variant %q which declares variants of its own; resolution is a single pass, so those would be silently ignored", entry.Name, v.Model) + } + + // Tags come from the REFERENCED entry, not from the declaring one: the + // serving feature being ranked is a property of the build that would be + // installed, and a parent's tags describe the model family. + option := VariantOption{Variant: v, Backend: target.Backend, Tags: target.Tags} + if env.ProbeMemory != nil { + option.ProbedMemory = env.ProbeMemory(target) + } + options = append(options, option) + } + + // The base is probed like any other candidate. It is never filtered out, but + // it IS ranked against the variants, and an unsized base would lose every + // contest to a variant whose size nothing could measure. + base := VariantOption{ + Variant: Variant{Model: entry.Name}, + Backend: entry.Backend, + IsBase: true, + Tags: entry.Tags, + } + if env.ProbeMemory != nil { + base.ProbedMemory = env.ProbeMemory(entry) + } + return append(options, base), nil +} + +// probeContextLength is the context size the footprint estimate is taken at. +// Selection compares whole models against whole hosts, so this only has to be a +// consistent, realistic default rather than the context a user will eventually +// configure. +const probeContextLength = 8192 + +// probeTimeout bounds a single entry's probe. An entry with several variants +// probes each of them, so an unreachable host has to give up quickly: an +// unknown size only costs a variant its ranking, whereas a stalled probe costs +// the user the whole install. +const probeTimeout = 5 * time.Second + +// probeEntryMemory measures what a gallery entry will occupy, without +// downloading it. +// +// pkg/vram does the work and caches its results across calls: it range-fetches +// the GGUF header for a real estimate, falls back to an HTTP HEAD for the +// content length, and finally to any declared size:. A HuggingFace repo listing +// is deliberately NOT used as a further fallback, unlike the gallery UI's +// estimator: a quantization entry's urls routinely point at the base model's +// repo, and summing every weight file there would overstate one variant badly +// enough to wrongly filter it out. +// +// Zero means the size could not be determined. Callers must read that as +// unknown rather than as zero, so an unreachable network downgrades a variant's +// ranking instead of failing the install. +func probeEntryMemory(ctx context.Context, entry *GalleryModel) uint64 { + input := vram.ModelEstimateInput{Size: entry.Size} + for _, f := range entry.AdditionalFiles { + if vram.IsWeightFile(f.URI) { + input.Files = append(input.Files, vram.FileInput{URI: f.URI}) + } + } + if len(input.Files) == 0 && input.Size == "" { + return 0 + } + + ctx, cancel := context.WithTimeout(ctx, probeTimeout) + defer cancel() + + estimate, err := vram.EstimateModelMultiContext(ctx, input, []uint32{probeContextLength}) + if err != nil { + xlog.Debug("Could not probe a model variant's size; treating it as unknown", "model", entry.Name, "error", err) + return 0 + } + return estimate.VRAMForContext(probeContextLength) +} + +// ResolveVariant picks the gallery entry to install for a host, from the +// variants an entry declares plus the entry itself, and returns it renamed to +// the entry's name and carrying the entry's presentation metadata. +// +// Why the metadata split: the payload (url, config_file, files, overrides) +// must come from the chosen variant because that is what actually gets +// downloaded, while the presentation (name, description, icon, tags) must come +// from the entry the user asked for, so the installed model presents as that +// model rather than as one of its variants. +// +// The base entry always resolves, whatever the host has. It is a complete entry +// that every older LocalAI release installs unconditionally, so refusing it here +// would make the gallery behave worse the newer the client is. Being exempt from +// the filters does not make it exempt from ranking: it is measured and compared +// like every declared variant, and wins by default only once they have all been +// ruled out. +func ResolveVariant(models []*GalleryModel, entry *GalleryModel, env ResolveEnv, pin string) (*GalleryModel, Variant, error) { + options, err := variantOptions(models, entry, env) + if err != nil { + return nil, Variant{}, err + } + + selection, err := SelectVariant(options, env, pin) + if err != nil { + return nil, Variant{}, fmt.Errorf("resolving variant for model %q: %w", entry.Name, err) + } + selected := selection.Option + + if selection.FellBackToBase && len(entry.Variants) > 0 { + xlog.Warn("No declared variant of this model fits this system; installing the entry's own build", + "model", entry.Name, "available_memory", env.AvailableMemory, "reasons", strings.Join(selection.Reasons, "; ")) + } + + // A pin is an operator override and deliberately bypasses the hardware + // checks, but a silent bypass makes a later out-of-memory failure + // impossible to trace back to the pin, so it is recorded loudly here. + if pin != "" { + if need, known := selected.EffectiveMemory(); known && env.AvailableMemory < need { + xlog.Warn("Pinned model variant declares more memory than this system reports; installing anyway because the pin overrides hardware resolution", + "model", entry.Name, "variant", selected.Variant.Model, "required_memory", need, "available_memory", env.AvailableMemory) + } + } + + // Selecting the base means installing the entry's own payload, which is the + // entry itself; there is no second entry to look up. + source := entry + if !selected.IsBase { + // variantOptions already proved this lookup succeeds. + source = FindGalleryElement(models, selected.Variant.Model) + } + + resolved := *source + resolved.Name = entry.Name + resolved.Description = entry.Description + resolved.Icon = entry.Icon + resolved.License = entry.License + // The resolved entry is a concrete install target, so it must not carry the + // variant list any more; leaving it would let a second pass resolve the + // already-resolved entry all over again. + resolved.Variants = nil + + // The struct copy above is shallow, so every reference-typed field still + // aliases the gallery's own entries. The install path mutates Overrides in + // place (mergo merges the caller's request overrides into it) and appends to + // the URL and tag slices, which would write the caller's request into the + // gallery catalog itself and leak between installs the moment this path + // reads from a cached, long-lived gallery listing. Detach them here. + // + // Overrides and ConfigFile are copied all the way down rather than cloned at + // the top level only: gallery overrides are nested in practice (a + // parameters.model map is near-universal), and mergo recurses into nested + // maps and overwrites them in place, so a top-level clone would still hand + // the caller the gallery's own inner maps. The slices below hold value types, + // so cloning them once fully detaches them. + resolved.Overrides = deepCopyStringMap(source.Overrides) + resolved.ConfigFile = deepCopyStringMap(source.ConfigFile) + resolved.AdditionalFiles = slices.Clone(source.AdditionalFiles) + resolved.URLs = slices.Clone(entry.URLs) + resolved.Tags = slices.Clone(entry.Tags) + + return &resolved, selected.Variant, nil +} + +// HostResolveEnv describes this machine to variant selection. +// +// It exists so the install path and every read-only surface that reports what +// selection WOULD choose derive the host from one place. Two copies of this +// wiring would eventually disagree, and a picker that shows a different answer +// than the installer produces is worse than no picker at all. +func HostResolveEnv(ctx context.Context, systemState *system.SystemState) ResolveEnv { + return ResolveEnv{ + AvailableMemory: availableModelMemory(systemState), + // The whole hardware gate. IsBackendCompatible already derives + // Darwin-only, NVIDIA-only, ROCm-only and SYCL-only from the backend + // name, so a gallery author never has to describe hardware. + // + // The uri argument is deliberately empty: it exists for backend OCI + // images, and passing a model's gallery url here would let an unrelated + // substring in a download link decide hardware compatibility. + BackendCompatible: func(backend string) bool { + return systemState.IsBackendCompatible(backend, "") + }, + // The ranking half of the hardware story. IsBackendCompatible only rules + // out what cannot run; on a Mac both an MLX and a llama.cpp build can, + // and this is what makes the native runtime win. + // + // EnginePreferenceTokens, NOT BackendPreferenceTokens: a variant is + // matched on its gallery `backend:` engine name, and the latter reports + // build tags that no engine name contains. + EnginePreference: systemState.EnginePreferenceTokens(), + // The third vocabulary, and the only one that is not host derived: no + // hardware prefers a plain build over an equivalent faster one, so this + // comes from a package function rather than from systemState. + ServingFeaturePreference: system.ServingFeaturePreferenceTokens(), + ProbeMemory: func(target *GalleryModel) uint64 { + return probeEntryMemory(ctx, target) + }, + } +} + +// noGPUDetected is what SystemState.DetectedCapability() reports when no +// usable GPU was found. The constant is unexported in pkg/system. +const noGPUDetected = "default" + +// availableModelMemory reports how much memory a model may occupy on this host. +// +// With a discrete GPU the model lives in VRAM. Otherwise it lives in system +// RAM, read through xsysinfo because that path is cgroup-aware: under +// Kubernetes the container's limit, not the node's physical RAM, is what the +// model actually gets. +// +// A detected GPU whose VRAM reads as zero falls back to RAM rather than to +// zero. That is the normal shape of a unified-memory host, not an error: +// arm64 macs report the metal capability unconditionally, yet have no discrete +// VRAM pool for TotalAvailableVRAM to find, so the model's real budget is +// system RAM. Returning the zero here would strand every Mac on the base +// build no matter how much memory it has. +// +// An unreadable RAM figure yields 0, which drops every variant with a known +// requirement and installs the base. That is the safe direction: an unknown +// host should not be talked into a larger download. +func availableModelMemory(systemState *system.SystemState) uint64 { + if systemState.DetectedCapability() != noGPUDetected && systemState.VRAM > 0 { + return systemState.VRAM + } + ram, err := xsysinfo.GetSystemRAMInfo() + if err != nil || ram == nil { + xlog.Warn("Could not read system RAM; treating this host as having no memory budget for variant selection", "error", err) + return 0 + } + return ram.Total +} + +// deepCopyStringMap copies a decoded YAML map so no part of the result, at any +// depth, is reachable from the original. +func deepCopyStringMap(m map[string]any) map[string]any { + if m == nil { + return nil + } + out := make(map[string]any, len(m)) + for k, v := range m { + out[k] = deepCopyYAMLValue(v) + } + return out +} + +// deepCopyYAMLValue recurses through the only container shapes a YAML decoder +// produces. Scalars are returned as-is because they cannot be mutated through +// the copy. map[any]any is handled as well because gopkg.in/yaml.v2 decodes +// non-string keys into it, and gallery documents are not guaranteed to have +// passed through the v3 decoder. +func deepCopyYAMLValue(v any) any { + switch t := v.(type) { + case map[string]any: + return deepCopyStringMap(t) + case map[any]any: + out := make(map[any]any, len(t)) + for k, val := range t { + out[k] = deepCopyYAMLValue(val) + } + return out + case []any: + out := make([]any, len(t)) + for i, val := range t { + out[i] = deepCopyYAMLValue(val) + } + return out + default: + return v + } +} + // Installs a model from the gallery func InstallModelFromGallery( ctx context.Context, @@ -81,7 +377,9 @@ func InstallModelFromGallery( modelLoader *model.ModelLoader, name string, req GalleryModel, downloadStatus func(string, string, string, float64), enforceScan, automaticallyInstallBackend, requireBackendIntegrity bool, options ...InstallOption) error { - applyModel := func(model *GalleryModel) error { + installOpts := applyInstallOptions(options...) + + applyModel := func(model *GalleryModel, record *ModelConfig) error { name = strings.ReplaceAll(name, string(os.PathSeparator), "__") var config ModelConfig @@ -104,13 +402,46 @@ func InstallModelFromGallery( ConfigFile: string(reYamlConfig), Description: model.Description, License: model.License, - URLs: model.URLs, Name: model.Name, Files: make([]File, 0), // Real values get added below, must be blank + // URLs are deliberately not seeded here: they are appended once + // below for both branches, and seeding them too would write every + // URL twice into the persisted gallery file. // Prompt Template Skipped for now - I expect in this mode that they will be delivered as files. } } else { - return fmt.Errorf("invalid gallery model %+v", model) + // An entry that names no base config describes itself entirely + // through overrides: and files:, so the base is simply empty. + // + // This is what the several hundred entries pointing at + // gallery/virtual.yaml were already getting. That stub carries only + // a name, a description and a license, all three of which are + // overwritten from the gallery entry a few lines up, and overrides + // reach InstallModel as a separate argument rather than being merged + // into the fetched config. So the fetch bought nothing beyond a + // round trip to GitHub on every install, and an author who omits the + // key is asking for exactly the same thing. + if !model.installsSomething(req) { + return fmt.Errorf("gallery model %q installs nothing: it declares no url, no config_file, no overrides and no files", model.Name) + } + config = ModelConfig{ + Description: model.Description, + License: model.License, + Name: model.Name, + Files: make([]File, 0), // Real values get added below, must be blank + // URLs are appended once below for every branch, as in the + // config_file case above. + } + } + + if record != nil { + config.EntryName = record.EntryName + config.ResolvedVariant = record.ResolvedVariant + config.PinnedVariant = record.PinnedVariant + // The variant's own name would otherwise be persisted here, which + // contradicts the whole point of variants: the model is known by + // the entry's stable name regardless of which variant backs it. + config.Name = record.EntryName } installName := model.Name @@ -157,7 +488,77 @@ func InstallModelFromGallery( return fmt.Errorf("no model found with name %q", name) } - return applyModel(model) + // An entry without variants is installed directly. An entry with them is + // still installable as-is; selection below only decides whether one of its + // declared alternatives suits this host better. + if !model.HasVariants() { + // A caller who named a variant asked for something this entry cannot + // give. Installing the entry anyway would look like success while + // quietly ignoring the choice, so it is refused by name instead. + if installOpts.variant != "" { + return fmt.Errorf("%w: %q was requested but model %q declares no variants", ErrPinNotFound, installOpts.variant, model.Name) + } + return applyModel(model, nil) + } + + pin := installOpts.variant + // A previously recorded pin survives reinstalls and upgrades, so a user + // who deliberately chose a variant is not silently re-resolved onto a + // different one by a hardware or gallery change. + // + // The record is keyed by the name the model was installed under, not by the + // gallery entry name: applyModel writes it to ._gallery_.yaml, + // where installName is req.Name whenever the caller supplied one. Reading it + // back under the entry's own name would miss the record for every custom-named + // install and silently re-resolve a deliberately pinned model onto a + // different variant, possibly swapping its backend. + // + // recalledPin tracks where the pin came from, because the two sources must + // fail differently when the name no longer resolves. See below. + recalledPin := "" + if pin == "" { + installName := model.Name + if req.Name != "" { + installName = req.Name + } + if previous, err := GetLocalModelConfiguration(systemState.Model.ModelsPath, installName); err == nil && previous != nil { + recalledPin = previous.PinnedVariant + pin = recalledPin + } + } + + env := HostResolveEnv(ctx, systemState) + + resolved, variant, err := ResolveVariant(models, model, env, pin) + // A pin the caller supplied on this request must stay fatal: they named + // something this entry cannot give, and installing anything else would + // report success for a request that was not honored. + // + // A pin recalled from disk is different. The gallery can rename or withdraw + // a variant long after it was pinned, and the user is not asking for it + // again on this call. Failing here would turn one gallery edit into a + // permanently unrepairable model whose only remedy is deleting a dotfile + // they have never heard of, so the stale pin is dropped and selection runs + // as if it had never been recorded. + if err != nil && recalledPin != "" && errors.Is(err, ErrPinNotFound) { + xlog.Warn("The recorded variant pin for this model no longer exists in the gallery; re-selecting automatically", + "model", model.Name, "dropped_pin", recalledPin) + pin = "" + resolved, variant, err = ResolveVariant(models, model, env, "") + } + if err != nil { + return err + } + + xlog.Info("Resolved model to variant", + "model", model.Name, "variant", variant.Model, + "available_memory", env.AvailableMemory, "pinned", pin != "") + + return applyModel(resolved, &ModelConfig{ + EntryName: model.Name, + ResolvedVariant: variant.Model, + PinnedVariant: pin, + }) } func InstallModel(ctx context.Context, systemState *system.SystemState, nameOverride string, galleryConfig *ModelConfig, configOverrides map[string]any, downloadStatus func(string, string, string, float64), enforceScan bool, options ...InstallOption) (*lconfig.ModelConfig, error) { diff --git a/core/gallery/models_types.go b/core/gallery/models_types.go index 24e32e03e80b..a623ff5354b6 100644 --- a/core/gallery/models_types.go +++ b/core/gallery/models_types.go @@ -15,6 +15,37 @@ type GalleryModel struct { ConfigFile map[string]any `json:"config_file,omitempty" yaml:"config_file,omitempty"` // Overrides are used to override the configuration of the model located at URL Overrides map[string]any `json:"overrides,omitempty" yaml:"overrides,omitempty"` + // Variants is an optional, UNORDERED list of alternative builds of the same + // model (other backends such as MLX or vLLM, other quantizations) that the + // installer may pick instead of this entry's own payload. Authoring is + // deliberately dumb: name the models, and the selector works out which one + // this host should get. + // + // The entry itself is always the last resort, so an entry carrying variants + // stays a complete, installable entry and older LocalAI releases, which drop + // this key, install it exactly as before. + Variants []Variant `json:"variants,omitempty" yaml:"variants,omitempty"` +} + +// installsSomething reports whether this entry, combined with the caller's +// request, would put anything on disk. +// +// It exists to keep an authoring mistake loud. Once an entry with no url and no +// config_file is accepted as an empty base, the only thing separating a +// deliberate overrides-only entry from a half-written stanza is whether it +// carries a payload at all. Without this check the stub would install cleanly +// and leave a model directory holding a config naming no weights, which fails +// far from the entry that caused it. +// +// The request is counted because its overrides and files are merged into the +// install exactly as the entry's own are, so a caller supplying them really has +// asked for something installable. +// +// An entry with a url or a config_file never reaches this: it has a base config +// to install, however thin. +func (m *GalleryModel) installsSomething(req GalleryModel) bool { + return len(m.Overrides) > 0 || len(m.AdditionalFiles) > 0 || + len(req.Overrides) > 0 || len(req.AdditionalFiles) > 0 } func (m *GalleryModel) GetInstalled() bool { @@ -45,6 +76,13 @@ func (m GalleryModel) ID() string { return fmt.Sprintf("%s@%s", m.Gallery.Name, m.Name) } +// HasVariants reports whether this entry declares alternative builds of itself. +// It says nothing about the entry being installable: an entry with variants is +// a complete entry that can always be installed as-is. +func (m GalleryModel) HasVariants() bool { + return len(m.Variants) > 0 +} + func (m *GalleryModel) GetTags() []string { return m.Tags } diff --git a/core/gallery/resolve_variant.go b/core/gallery/resolve_variant.go new file mode 100644 index 000000000000..e3f1b4699e86 --- /dev/null +++ b/core/gallery/resolve_variant.go @@ -0,0 +1,435 @@ +package gallery + +import ( + "errors" + "fmt" + "slices" + "sort" + "strings" +) + +var ( + // ErrNoVariantMatch is returned when nothing at all is selectable, which + // only happens for a caller that supplies no base option. + ErrNoVariantMatch = errors.New("no model variant matches this system") + // ErrPinNotFound is returned when a pinned variant is absent from the list. + ErrPinNotFound = errors.New("pinned model variant not found") +) + +// VariantOption is a variant paired with the facts a host is matched against. +// +// The install layer builds these by looking the referenced entries up in the +// live gallery. The selector never touches the catalog, which is what keeps it +// pure and testable across hardware shapes without touching the machine. +type VariantOption struct { + Variant Variant + // Backend is the engine the referenced entry resolves to (e.g. "llama-cpp", + // "mlx", "vllm"). Hardware support is derived from this name alone, which is + // precisely why a gallery author never has to describe hardware. + Backend string + // IsBase marks the declaring entry's own payload. It is exempt from every + // filter, because the entry must stay installable on every host and for + // every client, but it is otherwise an ordinary candidate: it is ranked + // against the declared variants rather than consulted only once they have + // all been rejected. + IsBase bool + // ProbedMemory is the footprint measured live from the referenced entry's + // weights, in bytes. It is the only source of a variant's size. An author + // who needs to correct a bad estimate sets `size:` on the referenced entry, + // which the estimator behind the probe already prefers over its own + // guesswork, so the correction lands everywhere the size is used. + // + // Zero means the probe could not determine a size. That is an unknown, not + // a zero requirement: a probe that cannot reach the network must never be + // able to break an install. + ProbedMemory uint64 + // Tags are the referenced entry's declared tags, and they are the SOLE + // serving feature signal. A tag is something an author wrote down on + // purpose, whereas an entry name only happens to contain a marker, so + // nothing else is consulted. + // + // Empty therefore means "declares no serving feature", and an entry that + // enables one without saying so ranks as a plain build. That is the cost of + // making the signal a declaration, and it is the right cost: an install + // silently downgraded to the plain build is recoverable, whereas a build + // promoted on a marker nobody meant is not visible at all. + // .agents/adding-gallery-models.md states the tagging rule. + Tags []string +} + +// EffectiveMemory returns this option's memory requirement in bytes and whether +// one is known at all. +func (o VariantOption) EffectiveMemory() (uint64, bool) { + if o.ProbedMemory > 0 { + return o.ProbedMemory, true + } + return 0, false +} + +// ResolveEnv describes the host a variant is selected for. +type ResolveEnv struct { + // AvailableMemory is what a model may occupy on this host: VRAM when a + // usable GPU was detected, system RAM otherwise. A model's footprint is + // roughly the same in either, so one number and one comparison cover both. + AvailableMemory uint64 + // BackendCompatible reports whether a backend can run here. The install + // layer wires SystemState.IsBackendCompatible, which already derives + // Darwin-only, NVIDIA-only, ROCm-only and SYCL-only from the backend name. + // + // A nil func treats every backend as runnable, the right default for a + // caller with no view of the hardware. + BackendCompatible func(backend string) bool + // EnginePreference lists the ENGINE NAMES this host prefers, best first, as + // SystemState.EnginePreferenceTokens reports them (e.g. NVIDIA gives + // ["vllm", "sglang", "llama-cpp"], metal gives ["mlx", "llama-cpp"]). A + // token is matched as a substring of a variant's backend name. + // + // The vocabulary is load bearing. VariantOption.Backend is a gallery entry's + // `backend:` value, which is an engine name and never carries a build tag, + // so build tags like "cuda" or "rocm" match NOTHING here. Do not wire + // SystemState.BackendPreferenceTokens into this field: that reports build + // tags for installed-build alias resolution, and the mismatch does not + // error, it scores every candidate equally and silently reduces selection to + // size alone. pkg/system/capabilities.go documents both tables. + // + // BackendCompatible answers "can this run here at all" and is a filter. + // This answers "which of the things that CAN run here should win" and is + // only a ranking. The two are deliberately separate: a Mac can run both an + // MLX build and a llama.cpp build, so nothing is filtered, yet the native + // accelerated runtime should still be installed even when the GGUF build is + // the larger download. + // + // An empty list ranks every backend equally, which reduces selection to + // ordering by size alone. That is the right default for a caller with no + // view of the hardware, and it is what every host looked like before + // preference existed. + EnginePreference []string + // ServingFeaturePreference lists the SERVING FEATURES to prefer, best first, + // as system.ServingFeaturePreferenceTokens reports them (currently + // ["dflash", "mtp"]). A token matches when it equals one of the variant's + // declared TAGS, compared whole and case-insensitively. Nothing else is + // matched: not the entry name, not its backend options. + // + // A third vocabulary, matched against a third thing. It is neither a build + // tag nor an engine name: these name a way of serving the same weights + // faster. pkg/system/capabilities.go documents all three tables together + // and justifies why this one reads a declaration rather than free text. + // + // An empty list ranks every build equally on this axis, which is what every + // host looked like before serving features were ranked at all. + ServingFeaturePreference []string + // ProbeMemory measures how much memory a referenced gallery entry needs, + // without downloading it. A zero result means "could not tell", never + // "needs nothing". + // + // It is a func field rather than a live network handle so specs can pin an + // exact size, or an exact failure, without reaching the internet. A nil func + // leaves every variant unknown, which selection already handles. + // + // SelectVariant never calls this: the install layer resolves every size into + // VariantOption.ProbedMemory first, so the selector stays pure. + ProbeMemory func(entry *GalleryModel) uint64 +} + +func (e ResolveEnv) backendRuns(backend string) bool { + if e.BackendCompatible == nil { + return true + } + return e.BackendCompatible(backend) +} + +// preferenceRank scores a backend against this host's preferred engine order, +// lower being better. +// +// It walks EnginePreference generically and names no engine and no capability: +// everything a new runtime needs is expressed by the table in pkg/system, so +// adding one never reaches this function. +// +// A backend matching no token scores just below the least preferred known one. +// It is never dropped, and a field where nothing matches scores uniformly, so +// an unrecognised backend, an unrecognised capability and an absent preference +// list all degrade to the same predictable place: ordering by size alone. +func (e ResolveEnv) preferenceRank(backend string) int { + name := strings.ToLower(backend) + return preferenceIndex(e.EnginePreference, func(token string) bool { + return strings.Contains(name, token) + }) +} + +// servingFeatureRank scores a variant against the preferred serving feature +// order, lower being better. +// +// It names no feature, exactly as preferenceRank names no engine: the ordered +// list comes from pkg/system, so teaching LocalAI about a new serving feature +// is a one-line edit to that table and never reaches this file. +// +// A declared TAG is the ONLY signal, compared whole and case-insensitively. An +// entry name is deliberately not consulted: a naming convention is not a +// contract, and an author is free to spell a build however they like, so +// reading a marker out of a name infers a capability nobody declared. Tags are +// a LocalAI-level vocabulary the gallery curators control and they mean the +// same thing on every backend. `overrides.options` was considered as the +// signal and rejected for the opposite reason: spec_type:draft-mtp is what +// actually enables the feature, but that spelling is llama.cpp's config +// vocabulary, and a cross-backend ranking decision must not depend on one +// backend's option syntax. Curators check the options at tagging time; the +// ranker reads only the tag. +// +// A build declaring no feature is a plain build and scores just below the +// least preferred known one, which is the whole point: whenever a faster way +// to serve the same weights survived the filters, it outranks the plain build. +func (e ResolveEnv) servingFeatureRank(o VariantOption) int { + tags := lowercased(o.Tags) + return preferenceIndex(e.ServingFeaturePreference, func(token string) bool { + return slices.Contains(tags, token) + }) +} + +// servingFeaturesOf lists the serving features a variant declares, best first. +// +// It is the descriptive half of servingFeatureRank: that function collapses the +// same match into one score for ordering, this one names what matched so a +// picker can tell the user WHY one build is preferred over another. Both read +// the single vocabulary on ResolveEnv, so a feature can never be shown as an +// advantage the ranker did not reward, nor rewarded without being shown. +// +// Preference order rather than the author's tag order, so the list reads best +// thing first. +// +// nil rather than an empty slice for a plain build, so the view field it feeds +// is omitted from the wire entirely. +func (e ResolveEnv) servingFeaturesOf(o VariantOption) []string { + if len(e.ServingFeaturePreference) == 0 || len(o.Tags) == 0 { + return nil + } + tags := lowercased(o.Tags) + var features []string + for _, token := range e.ServingFeaturePreference { + if slices.Contains(tags, strings.ToLower(strings.TrimSpace(token))) { + features = append(features, token) + } + } + return features +} + +// lowercased folds a tag list for comparison, so a gallery author writing +// "MTP" declares the same feature as one writing "mtp". +func lowercased(tags []string) []string { + if len(tags) == 0 { + return nil + } + out := make([]string, 0, len(tags)) + for _, t := range tags { + out = append(out, strings.ToLower(strings.TrimSpace(t))) + } + return out +} + +// preferenceIndex returns the position of the first token `matches` accepts. +// +// A subject matching no token scores just below the least preferred known one. +// It is never dropped, and a list where nothing matches scores uniformly, so an +// unrecognised subject and an absent preference list degrade to the same +// predictable place: this axis stops discriminating and the next key decides. +func preferenceIndex(tokens []string, matches func(token string) bool) int { + if len(tokens) == 0 { + return 0 + } + for i, token := range tokens { + if token != "" && matches(token) { + return i + } + } + return len(tokens) +} + +// VariantSelection is the outcome of a selection pass. +type VariantSelection struct { + Option VariantOption + // FellBackToBase reports that no declared variant survived the filters and + // the entry's own payload was all that remained. Callers log this, because a + // host quietly taking the base when upgrades were on offer is worth being + // able to see. + // + // It is deliberately narrower than "the base was selected": the base also + // wins on merit whenever it outranks every surviving variant, and that is an + // ordinary, uninteresting outcome rather than something to warn about. + FellBackToBase bool + // Reasons explains, one line per rejected variant, why it was dropped. + Reasons []string +} + +// SelectVariant picks the option to install for a host. +// +// The algorithm, in order: +// +// 1. An explicit pin wins outright, fit or no fit. It is a deliberate operator +// override, so refusing it would defeat the point; the caller warns. +// 2. Variants whose backend cannot run here are dropped. This is the whole of +// the hardware gate, derived from the backend name. +// 3. Variants whose known memory requirement exceeds what the host has are +// dropped. A variant with an UNKNOWN requirement survives, because nothing +// proves it does not fit and refusing on a size the probe could not read +// would let a network hiccup silently downgrade what gets installed. +// 4. The base survives both filters unconditionally, and then competes. It is +// a candidate like any other, not a last resort: an entry whose own build is +// the largest thing that fits must win against a smaller variant, and a base +// of known size must win against a variant whose size nothing could measure. +// 5. The survivors are ranked and the best one wins: fit tier first (rankOf +// below), then this host's engine preference, then the serving feature +// preference, then size. Both preferences beat size deliberately, so a Mac +// takes an MLX build over a larger GGUF one, and a host that can hold a +// speculative-decoding build of the same weights takes it over the plain +// one. +// 6. With no survivor at all, which can only happen when the caller supplied no +// base, there is nothing to install and this reports ErrNoVariantMatch. +func SelectVariant(options []VariantOption, env ResolveEnv, pin string) (VariantSelection, error) { + if pin != "" { + for _, o := range options { + if strings.EqualFold(o.Variant.Model, pin) { + return VariantSelection{Option: o}, nil + } + } + return VariantSelection{}, fmt.Errorf("%w: %q is not among the %d variants of this model", ErrPinNotFound, pin, len(options)) + } + + type ranked struct { + option VariantOption + memory uint64 + rank int + preference int + feature int + } + + survivors := make([]ranked, 0, len(options)) + reasons := make([]string, 0, len(options)) + survivingVariants := 0 + + for i := range options { + o := options[i] + memory, known := o.EffectiveMemory() + + // The base skips both gates. There is nothing below it, so refusing it + // would make an entry every older LocalAI installs fine uninstallable on + // newer ones. + if !o.IsBase { + if !env.backendRuns(o.Backend) { + reasons = append(reasons, fmt.Sprintf("%s needs backend %q, which cannot run on this system", o.Variant.Model, o.Backend)) + continue + } + if known && memory > env.AvailableMemory { + reasons = append(reasons, fmt.Sprintf("%s needs %s of memory", o.Variant.Model, humanBytes(memory))) + continue + } + survivingVariants++ + } + + survivors = append(survivors, ranked{ + option: o, + memory: memory, + rank: rankOf(o, env), + preference: env.preferenceRank(o.Backend), + feature: env.servingFeatureRank(o), + }) + } + + if len(survivors) == 0 { + return VariantSelection{}, fmt.Errorf( + "%w: %s of memory available; variants: %s", + ErrNoVariantMatch, humanBytes(env.AvailableMemory), strings.Join(reasons, "; "), + ) + } + + // Four keys, in this order, and the order is the whole design: + // + // 1. rank, the fit tier. Fit is a fact about whether the host can hold the + // build at all, so nothing outranks it. + // 2. preference, the host's runtime order. Among builds the host can + // equally hold, the one on the more native runtime wins even when it is + // the smaller download. A Mac given an MLX build and a larger llama.cpp + // build should run MLX; ranking by size first would install the GGUF and + // leave the accelerated build unused. Do NOT move this below size. + // 3. feature, the serving feature order. Among builds on an equally + // preferred runtime, one that speculates or predicts several tokens per + // step beats the plain build of the same weights, because it answers + // faster for the same output. + // + // Engine outranks this deliberately. A serving feature makes the right + // engine faster; it does not make a wrong engine right. Wearing the + // wrong engine costs the whole GPU, whereas a plain build on the native + // engine is merely slower than it could be, so a DFlash build on the + // portable engine must not beat a plain build on the engine this host + // actually serves well. + // 4. memory, largest first, because among builds on equally preferred + // runtimes with the same serving feature a bigger build is a higher + // quality quantization of the same model. Neither preference key may + // flatten this: two plain builds on one backend are still ordered by + // size. + // + // Stable so that options equal on all four keep their authored order, which + // is the only thing order still decides. + sort.SliceStable(survivors, func(i, j int) bool { + if survivors[i].rank != survivors[j].rank { + return survivors[i].rank < survivors[j].rank + } + if survivors[i].preference != survivors[j].preference { + return survivors[i].preference < survivors[j].preference + } + if survivors[i].feature != survivors[j].feature { + return survivors[i].feature < survivors[j].feature + } + return survivors[i].memory > survivors[j].memory + }) + + winner := survivors[0].option + return VariantSelection{ + Option: winner, + // Only a base that won by default is worth reporting. A base that + // outranked live competition is an ordinary selection. + FellBackToBase: winner.IsBase && survivingVariants == 0, + Reasons: reasons, + }, nil +} + +// Fit tiers, best first. Within a tier the host's preferred runtime wins, then +// the preferred serving feature, then the larger footprint; see the sort in +// SelectVariant. +const ( + // rankProvenFit is a measured size that the host is measured to satisfy. + rankProvenFit = iota + // rankBase is the entry's own build when it is not a proven fit: either it + // needs more memory than the host reports, or its size could not be + // measured either. It still outranks any unsized variant, because it is the + // payload the entry is guaranteed to be able to install and a variant of + // unmeasurable size is a guess. Taking the guess on a host that cannot be + // shown to accommodate it is how an unreachable network silently changes + // what gets installed. + rankBase + // rankUnknownFit is a variant whose size nothing could measure. Nothing + // proves it does not fit, so it is not dropped, but nothing proves it does + // either, so it ranks last. + rankUnknownFit +) + +func rankOf(o VariantOption, env ResolveEnv) int { + if memory, known := o.EffectiveMemory(); known && memory <= env.AvailableMemory { + return rankProvenFit + } + if o.IsBase { + return rankBase + } + return rankUnknownFit +} + +func humanBytes(b uint64) string { + const unit = 1024 + if b < unit { + return fmt.Sprintf("%dB", b) + } + div, exp := uint64(unit), 0 + for n := b / unit; n >= unit; n /= unit { + div *= unit + exp++ + } + return fmt.Sprintf("%.1f%ciB", float64(b)/float64(div), "KMGTPE"[exp]) +} diff --git a/core/gallery/resolve_variant_test.go b/core/gallery/resolve_variant_test.go new file mode 100644 index 000000000000..c5bd8648111e --- /dev/null +++ b/core/gallery/resolve_variant_test.go @@ -0,0 +1,1089 @@ +package gallery_test + +import ( + "context" + "os" + "path/filepath" + "runtime" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/mudler/LocalAI/core/gallery" + "github.com/mudler/LocalAI/pkg/system" +) + +var _ = Describe("VariantOption.EffectiveMemory", func() { + It("reports no requirement when nothing was probed", func() { + o := gallery.VariantOption{Variant: gallery.Variant{Model: "x"}} + size, known := o.EffectiveMemory() + Expect(known).To(BeFalse()) + Expect(size).To(Equal(uint64(0))) + }) + + It("uses the probed size", func() { + o := gallery.VariantOption{Variant: gallery.Variant{Model: "x"}, ProbedMemory: 6 * 1024 * 1024 * 1024} + size, known := o.EffectiveMemory() + Expect(known).To(BeTrue()) + Expect(size).To(Equal(uint64(6 * 1024 * 1024 * 1024))) + }) + + It("treats a failed probe as unknown rather than as a zero requirement", func() { + // A probe that could not reach the network reports 0. Reading that as + // "needs nothing" would make an unreachable host look like the perfect + // fit and hand the user the largest download on offer. + o := gallery.VariantOption{Variant: gallery.Variant{Model: "x"}, ProbedMemory: 0} + _, known := o.EffectiveMemory() + Expect(known).To(BeFalse()) + }) +}) + +var _ = Describe("SelectVariant", func() { + gib := func(n uint64) uint64 { return n * 1024 * 1024 * 1024 } + + // Every size here is a probed one, because the probe is now the only source + // a variant's footprint can come from. A zero stands for the probe having + // been unable to tell, which is an unknown rather than a zero requirement. + option := func(model, backend string, probed uint64) gallery.VariantOption { + return gallery.VariantOption{ + Variant: gallery.Variant{Model: model}, + Backend: backend, + ProbedMemory: probed, + } + } + + // The base is exempt from every filter, which the fallback specs below pin + // down, but it is ranked against the variants like any other candidate, so + // its size is load-bearing. + base := func(model string, probed uint64) gallery.VariantOption { + o := option(model, "llama-cpp", probed) + o.IsBase = true + return o + } + + // linuxNvidia mirrors what SystemState.IsBackendCompatible does on a Linux + // box with an NVIDIA card: Darwin-only engines are out, everything else runs. + linuxNvidia := func(backend string) bool { + return backend != "mlx" && backend != "mlx-vlm" + } + + Describe("hardware filtering", func() { + It("never selects a variant whose backend cannot run on this host", func() { + // The MLX build is both the largest and the only thing that would + // otherwise win, so nothing but the backend gate can reject it. + options := []gallery.VariantOption{ + option("m-mlx-8bit", "mlx", gib(24)), + option("m-gguf-q8", "llama-cpp", gib(12)), + base("m-gguf-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(80), + BackendCompatible: linuxNvidia, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-gguf-q8")) + }) + + It("selects the same variant on a host whose backend gate does admit it", func() { + // The mirror image of the spec above, so the rejection is proven to + // come from the host and not from something intrinsic to the entry. + options := []gallery.VariantOption{ + option("m-mlx-8bit", "mlx", gib(24)), + option("m-gguf-q8", "llama-cpp", gib(12)), + base("m-gguf-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(80), + BackendCompatible: func(string) bool { return true }, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-mlx-8bit")) + }) + + It("treats every backend as runnable when the host cannot be inspected", func() { + options := []gallery.VariantOption{option("m-mlx-8bit", "mlx", gib(24)), base("m-gguf-q4", gib(6))} + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(80)}, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-mlx-8bit")) + }) + }) + + Describe("ranking", func() { + It("picks the largest variant that fits, not the first authored", func() { + // Authored smallest-first, so first-match would take m-q4 and any + // ranking that ignores size would too. + options := []gallery.VariantOption{ + option("m-q4", "llama-cpp", gib(6)), + option("m-q8", "llama-cpp", gib(12)), + option("m-f16", "llama-cpp", gib(24)), + base("m-base", gib(2)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(16)}, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-q8")) + Expect(selection.FellBackToBase).To(BeFalse()) + }) + + It("picks the largest variant regardless of authored order", func() { + // Same set, authored largest-first. Order must make no difference at + // all, which is the entire point of dropping ordered first-match. + options := []gallery.VariantOption{ + option("m-f16", "llama-cpp", gib(24)), + option("m-q8", "llama-cpp", gib(12)), + option("m-q4", "llama-cpp", gib(6)), + base("m-base", gib(2)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(16)}, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-q8")) + }) + + It("prefers a known fit over a variant of unknown size", func() { + // An unknown requirement is a guess. It survives the filter, because + // nothing proves it does not fit, but it must never displace a + // variant that is known to fit. + options := []gallery.VariantOption{ + option("m-unknown", "llama-cpp", 0), + option("m-q8", "llama-cpp", gib(12)), + base("m-base", gib(2)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(16)}, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-q8")) + }) + + It("keeps a variant of unknown size rather than dropping it", func() { + options := []gallery.VariantOption{ + option("m-unknown", "llama-cpp", 0), + base("m-base", gib(2)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(4)}, "") + Expect(err).ToNot(HaveOccurred()) + // Surviving is observable through the rejection reasons: a dropped + // variant is always accounted for there, and this one is not. + Expect(selection.Reasons).ToNot(ContainElement(ContainSubstring("m-unknown"))) + // It survives, but it does not win: the base is a sized, guaranteed + // payload and an unmeasurable variant is a guess. + Expect(selection.Option.Variant.Model).To(Equal("m-base")) + Expect(selection.FellBackToBase).To(BeFalse()) + }) + + It("installs the base rather than an unsized variant on a host too small for either", func() { + // The exact shape 241 of the current index entries have: a referenced + // entry with no files and no size, whose probe can only answer + // "unknown". Ranking it above the base would install an unmeasured + // download on a machine with 2GiB, and would do so silently. + options := []gallery.VariantOption{ + option("m-unknown", "llama-cpp", 0), + base("m-base-q4", gib(4)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(2)}, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-base-q4")) + Expect(selection.Option.IsBase).To(BeTrue()) + }) + + It("selects the base when the base is the largest option that fits", func() { + // A Q8 base offering a Q4 downgrade for small hosts is a natural + // authoring shape. Treating the base as a last resort would install + // the Q4 on every host large enough for the Q8 and permanently + // downgrade the user. + options := []gallery.VariantOption{ + option("m-q4", "llama-cpp", gib(4)), + base("m-base-q8", gib(8)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(16)}, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-base-q8")) + // The Q4 survived every filter, so this is the base winning on rank + // and not the base being fallen back to. + Expect(selection.FellBackToBase).To(BeFalse()) + Expect(selection.Reasons).To(BeEmpty()) + }) + + It("selects a smaller variant when the base does not fit but the variant does", func() { + // The mirror of the spec above: the base competes, it does not win by + // default, so a host that cannot hold it must still take the downgrade + // the entry offers for exactly that case. + options := []gallery.VariantOption{ + option("m-q4", "llama-cpp", gib(4)), + base("m-base-q8", gib(8)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(6)}, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-q4")) + }) + + It("reports why a probed size that does not fit was rejected", func() { + // Ranking and filtering both run off the probe, so a rejection has to + // be traceable back to the figure the probe returned. + options := []gallery.VariantOption{ + option("m-q4", "llama-cpp", gib(6)), + option("m-f16", "llama-cpp", gib(24)), + option("m-q8", "llama-cpp", gib(12)), + base("m-base", gib(2)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(16)}, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-q8")) + Expect(selection.Reasons).To(ContainElement(ContainSubstring("m-f16"))) + }) + + It("admits a variant needing exactly the memory available", func() { + options := []gallery.VariantOption{option("m-q8", "llama-cpp", gib(12)), base("m-base", gib(2))} + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(12)}, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-q8")) + }) + }) + + Describe("ranking by host engine preference", func() { + // These are ENGINE NAMES, exactly what SystemState.EnginePreferenceTokens + // reports for these hosts and exactly what a gallery entry's `backend:` + // field holds. Build tags ("cuda", "rocm", "metal") belong to + // BackendPreferenceTokens and would match no engine name here, which is + // why every backend below is spelled as a real gallery engine. + // + // They are spelled out rather than read from the live machine so the + // specs pin the intended behaviour on every CI runner. + darwinMetal := []string{"mlx", "llama-cpp"} + nvidia := []string{"vllm", "sglang", "llama-cpp"} + + // A Mac runs both engines, so nothing is filtered here and preference is + // the only thing that can decide. + darwinRunsEverything := func(string) bool { return true } + + It("prefers a vLLM build to a larger llama.cpp build on nvidia", func() { + // The rule the engine table exists to express, and the one that was + // silently inert while the ranker was fed build tags: no gallery + // engine name contains "cuda", so every candidate scored equal and + // the larger llama.cpp build won. Emptying the nvidia rule in + // pkg/system fails this spec. + options := []gallery.VariantOption{ + option("m-vllm-awq", "vllm", gib(8)), + option("m-gguf-q8", "llama-cpp", gib(24)), + base("m-gguf-q4", gib(4)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(64), + EnginePreference: nvidia, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-vllm-awq")) + }) + + It("takes the larger llama.cpp build on the same nvidia host once preference is unknown", func() { + // The mirror of the spec above, proving the vLLM win comes from the + // preference list and not from anything intrinsic to the option set. + // This is the state the whole feature was stuck in before the two + // vocabularies were separated. + options := []gallery.VariantOption{ + option("m-vllm-awq", "vllm", gib(8)), + option("m-gguf-q8", "llama-cpp", gib(24)), + base("m-gguf-q4", gib(4)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(64), + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-gguf-q8")) + }) + + It("ranks a vllm-omni build with vllm, since the token is a substring", func() { + // Substring matching is deliberate: vllm-omni is a vLLM build and + // must inherit vLLM's rank rather than fall through to unranked. + options := []gallery.VariantOption{ + option("m-omni", "vllm-omni", gib(8)), + option("m-gguf-q8", "llama-cpp", gib(24)), + base("m-base", gib(4)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(64), + EnginePreference: nvidia, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-omni")) + }) + + It("prefers an MLX build to a larger llama.cpp build on darwin", func() { + options := []gallery.VariantOption{ + option("m-mlx-4bit", "mlx", gib(8)), + option("m-gguf-q8", "llama-cpp", gib(24)), + base("m-gguf-q4", gib(4)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(64), + BackendCompatible: darwinRunsEverything, + EnginePreference: darwinMetal, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-mlx-4bit")) + }) + + It("takes the larger llama.cpp build on the same darwin host once preference is unknown", func() { + // The mirror of the spec above, proving the MLX win comes from the + // preference list and not from anything intrinsic to the option set. + options := []gallery.VariantOption{ + option("m-mlx-4bit", "mlx", gib(8)), + option("m-gguf-q8", "llama-cpp", gib(24)), + base("m-gguf-q4", gib(4)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(64), + BackendCompatible: darwinRunsEverything, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-gguf-q8")) + }) + + It("still picks the largest fitting build among equally preferred engines", func() { + // Preference must not flatten size ordering: with one engine in + // play there is nothing left for it to decide. + options := []gallery.VariantOption{ + option("m-gguf-q4", "llama-cpp", gib(6)), + option("m-gguf-q8", "llama-cpp", gib(12)), + option("m-gguf-f16", "llama-cpp", gib(48)), + base("m-base", gib(2)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(16), + EnginePreference: nvidia, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-gguf-q8")) + }) + + It("does not let a preferred engine rescue a build that does not fit", func() { + // Fit is a filter and preference is only a ranking among survivors. + // The vLLM build is both preferred and too large, so the llama.cpp + // build the host can actually hold has to win. + options := []gallery.VariantOption{ + option("m-vllm-fp16", "vllm", gib(48)), + option("m-gguf-q4", "llama-cpp", gib(6)), + base("m-base", gib(2)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(16), + EnginePreference: nvidia, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-gguf-q4")) + Expect(selection.Reasons).To(ContainElement(ContainSubstring("m-vllm-fp16"))) + }) + + It("orders engines absent from the table by size rather than dropping them", func() { + // Neither engine appears in the nvidia rule. Nothing is discarded + // and nothing is arbitrarily favoured; the host falls back to the + // size-only behaviour it had before preference existed. + // + // The base is left unsized so it ranks in the tier below a proven + // fit. It is a llama.cpp build, which the nvidia rule DOES rank, and + // letting it compete in the same tier would prove preference works + // rather than proving unlisted engines degrade to size. + options := []gallery.VariantOption{ + option("m-diffusers", "diffusers", gib(12)), + option("m-transformers", "transformers", gib(6)), + base("m-base", 0), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(16), + EnginePreference: nvidia, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-diffusers")) + Expect(selection.Reasons).To(BeEmpty()) + }) + + It("ranks sglang between vllm and llama-cpp on nvidia", func() { + // A judgement call worth pinning: sglang is a GPU serving engine of + // the same class as vllm, so it outranks the portable engine, but it + // sits behind vllm. + options := []gallery.VariantOption{ + option("m-vllm", "vllm", gib(4)), + option("m-sglang", "sglang", gib(6)), + option("m-gguf", "llama-cpp", gib(8)), + base("m-base", gib(2)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(16), + EnginePreference: nvidia, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-vllm")) + + withoutVLLM := []gallery.VariantOption{ + option("m-sglang", "sglang", gib(6)), + option("m-gguf", "llama-cpp", gib(8)), + base("m-base", gib(2)), + } + selection, err = gallery.SelectVariant(withoutVLLM, gallery.ResolveEnv{ + AvailableMemory: gib(16), + EnginePreference: nvidia, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-sglang")) + }) + + It("still installs the base when nothing fits, whatever the host prefers", func() { + options := []gallery.VariantOption{ + option("m-mlx-8bit", "mlx", gib(48)), + option("m-gguf-q8", "llama-cpp", gib(24)), + base("m-base", gib(64)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(4), + BackendCompatible: darwinRunsEverything, + EnginePreference: darwinMetal, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-base")) + Expect(selection.Option.IsBase).To(BeTrue()) + Expect(selection.FellBackToBase).To(BeTrue()) + }) + + It("honors a pin for a less preferred backend", func() { + // A pin is an operator override, so preference must not quietly + // redirect it any more than the memory filter does. + options := []gallery.VariantOption{ + option("m-mlx-4bit", "mlx", gib(8)), + option("m-gguf-q8", "llama-cpp", gib(24)), + base("m-base", gib(4)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(64), + BackendCompatible: darwinRunsEverything, + EnginePreference: darwinMetal, + }, "m-gguf-q8") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-gguf-q8")) + }) + }) + + Describe("ranking by serving feature", func() { + // These are SERVING FEATURES, exactly what + // system.ServingFeaturePreferenceTokens reports, and a third vocabulary + // after build tags and engine names. They are matched against a + // variant's declared TAGS and against nothing else. + // + // Spelled out rather than read from pkg/system so these specs pin the + // intended ordering rather than restating whatever the table says. + features := []string{"dflash", "mtp"} + nvidia := []string{"vllm", "sglang", "llama-cpp"} + + // A tag is the whole signal, so every candidate that is meant to carry + // a feature declares it here. Candidates built with `option` carry no + // tags and are therefore plain builds no matter what their name says, + // which several specs below rely on. + tagged := func(model string, probed uint64, tags ...string) gallery.VariantOption { + o := option(model, "llama-cpp", probed) + o.Tags = tags + return o + } + + It("prefers a speculative build to the plain build of the same weights", func() { + // The rule the feature table exists to express. Both builds fit and + // both run on the same engine, and the plain build is deliberately + // the larger one, so without this axis size would take it and the + // drafter pairing would never be installed. Emptying + // ServingFeaturePreference fails this spec. + options := []gallery.VariantOption{ + tagged("m-turbo-q4", gib(14), "llm", "dflash"), + tagged("m-q8", gib(20), "llm"), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(32), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-turbo-q4")) + }) + + It("takes the larger plain build once the feature preference is unknown", func() { + // The mirror of the spec above on the identical option set, proving + // the DFlash win comes from the feature list rather than from + // anything intrinsic to the set. + options := []gallery.VariantOption{ + tagged("m-turbo-q4", gib(14), "llm", "dflash"), + tagged("m-q8", gib(20), "llm"), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(32), + EnginePreference: nvidia, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-q8")) + }) + + It("prefers dflash to mtp", func() { + // Both are speculative pairings, so only the table's order can + // separate them, and the MTP build is deliberately the larger one so + // size cannot be what decides. + options := []gallery.VariantOption{ + tagged("m-alpha-q8", gib(20), "llm", "mtp"), + tagged("m-beta-q4", gib(14), "llm", "dflash"), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(32), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-beta-q4")) + }) + + It("does not let a preferred feature rescue a build that does not fit", func() { + // A drafter pairing is strictly larger than the plain build, so this + // is the ordinary case on a small host rather than a corner one. Fit + // is a filter; the feature preference only ranks survivors. + options := []gallery.VariantOption{ + tagged("m-turbo-f16", gib(48), "llm", "dflash"), + tagged("m-q8", gib(12), "llm"), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(16), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-q8")) + Expect(selection.Reasons).To(ContainElement(ContainSubstring("m-turbo-f16"))) + }) + + It("lets the host engine preference outrank the serving feature", func() { + // A serving feature makes the right engine faster; it does not make + // a wrong engine right. On nvidia the plain vLLM build therefore + // beats a DFlash llama.cpp build even though both fit and the + // llama.cpp one is larger. + options := []gallery.VariantOption{ + tagged("m-turbo-q8", gib(24), "llm", "dflash"), + option("m-vllm-awq", "vllm", gib(8)), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(64), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-vllm-awq")) + }) + + It("still picks the largest fitting build among equally featured builds", func() { + // Neither preference key may flatten size ordering: with one engine + // and one feature in play there is nothing left for them to decide. + options := []gallery.VariantOption{ + tagged("m-turbo-q4", gib(8), "llm", "dflash"), + tagged("m-turbo-q8", gib(14), "llm", "dflash"), + tagged("m-turbo-f16", gib(48), "llm", "dflash"), + base("m-q4", gib(2)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(16), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-turbo-q8")) + }) + + It("leaves an unfeatured build ranked last rather than dropping it", func() { + // Ranking never filters. With only plain builds on offer the axis + // scores them uniformly and selection degrades to size alone. + options := []gallery.VariantOption{ + option("m-q8", "llama-cpp", gib(12)), + option("m-q4", "llama-cpp", gib(6)), + base("m-q2", gib(3)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(32), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-q8")) + Expect(selection.Reasons).To(BeEmpty()) + }) + + Describe("reading the declared tags", func() { + It("prefers a build whose tag declares the feature its name does not", func() { + // The case tags exist for. "m-turbo-q8" carries MTP heads but + // spells nothing in its name, which is the shape most of the + // gallery's MTP entries had before they were tagged. It is also + // the smaller build, so only the feature axis can lift it. + options := []gallery.VariantOption{ + tagged("m-turbo-q8", gib(14), "llm", "gguf", "mtp"), + tagged("m-plain-q8", gib(20), "llm", "gguf"), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(32), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-turbo-q8")) + }) + + It("does NOT prefer a build that only its name declares, with no tags at all", func() { + // The inversion of the old name-fallback spec, and the + // regression this change exists to guard. A name is + // author-supplied free text and a naming convention is not a + // contract: "m-nvfp4-mtp" is exactly the shape of the gallery's + // NVFP4 entries, whose weights carry MTP heads while the entry + // enables no speculative decoding at all, so ranking it as a + // speculative build made it win the feature axis without being + // any faster. With no tag it is a plain build and the larger + // plain build takes it on size. + options := []gallery.VariantOption{ + option("m-nvfp4-mtp", "llama-cpp", gib(14)), + tagged("m-plain-q8", gib(20), "llm", "gguf"), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(32), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-plain-q8")) + }) + + It("lets a tagged build beat a larger one whose name says the same feature", func() { + // The sharper form of the spec above: the name half is not + // merely unnecessary, it is not consulted. The untagged + // "m-dflash" would outrank the tagged MTP build on the old + // name-first ordering, and is the larger build besides, so it + // wins on either of the two ways the name could still be read. + options := []gallery.VariantOption{ + option("m-dflash", "llama-cpp", gib(20)), + tagged("m-turbo-q4", gib(14), "llm", "mtp"), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(32), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-turbo-q4")) + }) + + It("matches a tag regardless of the case it was written in", func() { + // Gallery tags are author-supplied, so the same declaration + // arrives in whatever case the author typed. A curator who + // writes "MTP" has declared the feature just as plainly as one + // who writes "mtp", and the sole signal must not turn on that. + options := []gallery.VariantOption{ + tagged("m-turbo-q8", gib(14), "LLM", "MTP"), + tagged("m-plain-q8", gib(20), "llm", "gguf"), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(32), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-turbo-q8")) + }) + + It("prefers dflash to mtp when both are declared by tag", func() { + // The table's order has to survive on the tag path with both + // features spelled out explicitly, and the MTP build is + // deliberately the larger one so size cannot be what decides. + options := []gallery.VariantOption{ + tagged("m-alpha-q8", gib(20), "llm", "mtp"), + tagged("m-beta-q8", gib(14), "llm", "dflash"), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(32), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-beta-q8")) + }) + + It("does not read a feature out of an unrelated tag", func() { + // Tags are compared whole, so a tag that merely contains a + // feature token declares nothing. "multimodal" contains no + // feature; "smtp" does contain "mtp" as a substring and must + // not count either. The mail model is the larger build, so a + // false positive would hand it the win. + options := []gallery.VariantOption{ + tagged("m-mail-q8", gib(24), "llm", "multimodal", "smtp"), + tagged("m-turbo-q4", gib(14), "llm", "mtp"), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(64), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-turbo-q4")) + }) + + It("does not let a tag rescue a build that does not fit", func() { + // Fit still outranks the feature axis, whichever signal + // declared it. + options := []gallery.VariantOption{ + tagged("m-turbo-f16", gib(48), "llm", "mtp"), + tagged("m-plain-q8", gib(12), "llm"), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(16), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-plain-q8")) + Expect(selection.Reasons).To(ContainElement(ContainSubstring("m-turbo-f16"))) + }) + + It("lets the host engine preference outrank a tagged serving feature", func() { + // Engine beats feature no matter which signal carried the + // feature: a tag does not make a wrong engine right. + options := []gallery.VariantOption{ + tagged("m-turbo-q8", gib(24), "llm", "mtp"), + option("m-vllm-awq", "vllm", gib(8)), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(64), + EnginePreference: nvidia, + ServingFeaturePreference: features, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-vllm-awq")) + }) + + It("ignores tags entirely when no feature preference is configured", func() { + // The axis stops discriminating with an empty list, tags or no + // tags, and selection degrades to size alone as it always did. + options := []gallery.VariantOption{ + tagged("m-turbo-q8", gib(14), "llm", "mtp"), + tagged("m-plain-q8", gib(20), "llm"), + base("m-q4", gib(6)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(32), + EnginePreference: nvidia, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-plain-q8")) + }) + }) + }) + + Describe("falling back to the base", func() { + It("selects the base when nothing else fits", func() { + options := []gallery.VariantOption{ + option("m-q8", "llama-cpp", gib(12)), + option("m-f16", "llama-cpp", gib(24)), + base("m-base", gib(2)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(4)}, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-base")) + Expect(selection.Option.IsBase).To(BeTrue()) + Expect(selection.FellBackToBase).To(BeTrue()) + Expect(selection.Reasons).To(HaveLen(2)) + }) + + It("selects the base even when the base does not fit either", func() { + // The base is exempt from the memory filter, not merely favoured by + // it: there is nothing below it, so refusing here would make an entry + // every older client installs fine uninstallable on newer ones. + options := []gallery.VariantOption{option("m-q8", "llama-cpp", gib(12)), base("m-base", gib(2))} + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: 0}, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-base")) + Expect(selection.Option.IsBase).To(BeTrue()) + Expect(selection.FellBackToBase).To(BeTrue()) + }) + + It("prefers the base to an unsized variant even when the base itself is unsized", func() { + // Neither can be shown to fit, so nothing separates them on size. The + // base is still the payload the entry is guaranteed to install. + options := []gallery.VariantOption{ + option("m-unknown", "llama-cpp", 0), + base("m-base", 0), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(8)}, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-base")) + }) + + It("explains why each variant was rejected", func() { + options := []gallery.VariantOption{ + option("m-mlx", "mlx", gib(8)), + option("m-f16", "llama-cpp", gib(24)), + base("m-base", gib(2)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(4), + BackendCompatible: linuxNvidia, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Reasons).To(ContainElement(ContainSubstring("cannot run on this system"))) + Expect(selection.Reasons).To(ContainElement(ContainSubstring("24.0GiB"))) + }) + + It("errors when the caller supplies no base at all", func() { + options := []gallery.VariantOption{option("m-f16", "llama-cpp", gib(24))} + + _, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(4)}, "") + Expect(err).To(MatchError(gallery.ErrNoVariantMatch)) + }) + }) + + Describe("explicit selection", func() { + It("honors a pin the hardware would never have chosen", func() { + options := []gallery.VariantOption{ + option("m-mlx", "mlx", gib(64)), + base("m-base", gib(2)), + } + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{ + AvailableMemory: gib(4), + BackendCompatible: linuxNvidia, + }, "m-mlx") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-mlx")) + }) + + It("honors a pin naming the base, declining an upgrade that fits", func() { + options := []gallery.VariantOption{option("m-f16", "llama-cpp", gib(8)), base("m-base", gib(2))} + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(64)}, "m-base") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-base")) + }) + + It("matches a pin case-insensitively", func() { + options := []gallery.VariantOption{option("m-f16", "llama-cpp", gib(8)), base("m-base", gib(2))} + + selection, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(64)}, "M-F16") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-f16")) + }) + + It("fails loudly when the pin names nothing in the list", func() { + options := []gallery.VariantOption{option("m-f16", "llama-cpp", gib(8)), base("m-base", gib(2))} + + _, err := gallery.SelectVariant(options, gallery.ResolveEnv{AvailableMemory: gib(64)}, "m-gone") + Expect(err).To(MatchError(gallery.ErrPinNotFound)) + Expect(err.Error()).To(ContainSubstring("m-gone")) + }) + }) +}) + +var _ = Describe("HostResolveEnv engine preference wiring", func() { + // The specs above feed SelectVariant a hand-written token list, which proves + // the ranker but not that the host actually reaches the engine table. These + // drive the REAL table through the REAL wiring, so emptying + // engineNamePreferenceRules in pkg/system, or reverting this field to + // BackendPreferenceTokens, fails here. + var origEnv, origRunFileEnv string + const capabilityEnv = "LOCALAI_FORCE_META_BACKEND_CAPABILITY" + const capabilityRunFileEnv = "LOCALAI_FORCE_META_BACKEND_CAPABILITY_RUN_FILE" + // What getSystemCapabilities reports for a host with no usable accelerator. + const noGPUCapability = "default" + + BeforeEach(func() { + origEnv = os.Getenv(capabilityEnv) + origRunFileEnv = os.Getenv(capabilityRunFileEnv) + }) + + AfterEach(func() { + if origEnv != "" { + Expect(os.Setenv(capabilityEnv, origEnv)).To(Succeed()) + } else { + Expect(os.Unsetenv(capabilityEnv)).To(Succeed()) + } + if origRunFileEnv != "" { + Expect(os.Setenv(capabilityRunFileEnv, origRunFileEnv)).To(Succeed()) + } else { + Expect(os.Unsetenv(capabilityRunFileEnv)).To(Succeed()) + } + }) + + envFor := func(capability string) gallery.ResolveEnv { + GinkgoHelper() + Expect(os.Setenv(capabilityEnv, capability)).To(Succeed()) + return gallery.HostResolveEnv(context.Background(), &system.SystemState{}) + } + + It("hands the ranker engine names an NVIDIA host's gallery entries can match", func() { + preference := envFor("nvidia-cuda-12").EnginePreference + Expect(preference).To(Equal([]string{"vllm", "sglang", "llama-cpp"})) + }) + + It("installs the vLLM build over a larger llama.cpp one on a real NVIDIA host", func() { + // End to end on the live table: the exact behaviour that was silently + // dead while the ranker was fed build tags. + env := envFor("nvidia-cuda-12") + env.AvailableMemory = 64 * 1024 * 1024 * 1024 + + options := []gallery.VariantOption{ + {Variant: gallery.Variant{Model: "m-vllm"}, Backend: "vllm", ProbedMemory: 8 * 1024 * 1024 * 1024}, + {Variant: gallery.Variant{Model: "m-gguf"}, Backend: "llama-cpp", ProbedMemory: 24 * 1024 * 1024 * 1024}, + } + + selection, err := gallery.SelectVariant(options, env, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-vllm")) + }) + + It("installs the MLX build over a larger llama.cpp one on a real darwin host", func() { + env := envFor("metal") + env.AvailableMemory = 64 * 1024 * 1024 * 1024 + // The real gate rejects mlx off darwin, and this spec is about ranking. + env.BackendCompatible = func(string) bool { return true } + + options := []gallery.VariantOption{ + {Variant: gallery.Variant{Model: "m-mlx"}, Backend: "mlx", ProbedMemory: 8 * 1024 * 1024 * 1024}, + {Variant: gallery.Variant{Model: "m-gguf"}, Backend: "llama-cpp", ProbedMemory: 24 * 1024 * 1024 * 1024}, + } + + selection, err := gallery.SelectVariant(options, env, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-mlx")) + }) + + It("installs the llama.cpp build over a larger vLLM one on a host with no GPU", func() { + // The hole this rule closes. IsBackendCompatible keys on the engine + // name, and "vllm" carries no darwin/cuda/rocm/sycl token, so a vLLM + // build is NOT filtered out here. Emptying the default rule in + // pkg/system puts the larger vLLM build back on a CPU-only box. + env := envFor(noGPUCapability) + env.AvailableMemory = 64 * 1024 * 1024 * 1024 + + options := []gallery.VariantOption{ + {Variant: gallery.Variant{Model: "m-vllm"}, Backend: "vllm", ProbedMemory: 24 * 1024 * 1024 * 1024}, + {Variant: gallery.Variant{Model: "m-gguf"}, Backend: "llama-cpp", ProbedMemory: 8 * 1024 * 1024 * 1024}, + } + + selection, err := gallery.SelectVariant(options, env, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-gguf")) + }) + + It("installs the llama.cpp build over a larger vLLM one on an intel mac", func() { + env := envFor("darwin-x86") + env.AvailableMemory = 64 * 1024 * 1024 * 1024 + + options := []gallery.VariantOption{ + {Variant: gallery.Variant{Model: "m-vllm"}, Backend: "vllm", ProbedMemory: 24 * 1024 * 1024 * 1024}, + {Variant: gallery.Variant{Model: "m-gguf"}, Backend: "llama-cpp", ProbedMemory: 8 * 1024 * 1024 * 1024}, + } + + selection, err := gallery.SelectVariant(options, env, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-gguf")) + }) + + It("still installs a vLLM build on a host with no GPU when it is the only one offered", func() { + // Preference ORDERS survivors, it never filters them. Demoting vLLM must + // not make a model published only as a vLLM build uninstallable. + env := envFor(noGPUCapability) + env.AvailableMemory = 64 * 1024 * 1024 * 1024 + + options := []gallery.VariantOption{ + {Variant: gallery.Variant{Model: "m-vllm"}, Backend: "vllm", ProbedMemory: 8 * 1024 * 1024 * 1024}, + } + + selection, err := gallery.SelectVariant(options, env, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-vllm")) + }) + + It("prefers llama.cpp on a GPU host with too little VRAM to serve from", func() { + // This host has a GPU, yet getSystemCapabilities reports "default" + // because it is under the 4 GiB floor. Driven through the real detector + // rather than a forced capability, so that mapping is exercised too. + if runtime.GOOS == "darwin" { + Skip("darwin reports metal or darwin-x86 before the VRAM floor is consulted") + } + Expect(os.Unsetenv(capabilityEnv)).To(Succeed()) + // A capability run file on the machine would override detection. + Expect(os.Setenv(capabilityRunFileEnv, filepath.Join(GinkgoT().TempDir(), "absent"))).To(Succeed()) + + state := &system.SystemState{GPUVendor: system.Nvidia, VRAM: 2 * 1024 * 1024 * 1024} + Expect(state.DetectedCapability()).To(Equal(noGPUCapability)) + + env := gallery.HostResolveEnv(context.Background(), state) + env.AvailableMemory = 64 * 1024 * 1024 * 1024 + + options := []gallery.VariantOption{ + {Variant: gallery.Variant{Model: "m-vllm"}, Backend: "vllm", ProbedMemory: 24 * 1024 * 1024 * 1024}, + {Variant: gallery.Variant{Model: "m-gguf"}, Backend: "llama-cpp", ProbedMemory: 8 * 1024 * 1024 * 1024}, + } + + selection, err := gallery.SelectVariant(options, env, "") + Expect(err).ToNot(HaveOccurred()) + Expect(selection.Option.Variant.Model).To(Equal("m-gguf")) + }) + + It("carries no build tag into the ranker, whatever the host", func() { + // The wiring-level lock. BackendPreferenceTokens returns build tags for + // every capability, so if it is ever wired back into this field, one of + // these tags shows up here. + for _, capability := range []string{"nvidia-cuda-12", "amd", "intel", "metal", "vulkan", "default"} { + Expect(envFor(capability).EnginePreference).ToNot( + ContainElements("cuda", "rocm", "hip", "sycl", "metal", "cpu", "darwin-x86"), + "capability %q leaked a build tag into the variant ranker", capability) + } + }) +}) diff --git a/core/gallery/variant.go b/core/gallery/variant.go new file mode 100644 index 000000000000..ad7ff093263b --- /dev/null +++ b/core/gallery/variant.go @@ -0,0 +1,18 @@ +package gallery + +// Variant is one option in a gallery entry's variant list. It references an +// existing gallery entry by name, and that is all an author has to write. +// +// Authored order carries no meaning. Selection filters out what this host +// cannot run and then ranks what is left, so hardware knowledge lives in the +// selector rather than being pushed onto whoever edits the gallery. +// +// There is deliberately no authored memory figure here. When the live probe +// misreads a variant's footprint, the fix is the referenced entry's own `size:` +// field: the estimator already prefers a declared size over its own guesswork, +// so correcting it there fixes the figure for every consumer rather than only +// for this one selection pass. +type Variant struct { + // Model is the name of a gallery entry that declares no variants of its own. + Model string `json:"model" yaml:"model"` +} diff --git a/core/gallery/variant_quantization.go b/core/gallery/variant_quantization.go new file mode 100644 index 000000000000..176620d487a9 --- /dev/null +++ b/core/gallery/variant_quantization.go @@ -0,0 +1,147 @@ +package gallery + +import ( + "path" + "regexp" + "strings" +) + +// quantizationToken matches one filename segment that names a weight format. +// +// The alternatives, in order: the GGUF k-quant/legacy family with any number of +// trailing qualifiers (`q8_0`, `q4_k_m`, `iq2_xxs`, `pq2_0`, `q2_0_g128`), the +// float formats (`f16`, `fp16`, `bf16`, `f32`) including the vendor 4-bit ones +// (`nvfp4`, `mxfp4`), and the bit-count spellings the MLX and AWQ ecosystems +// use (`4bit`, `int8`). +// +// It is anchored at both ends because it is applied to an already-split +// segment. Matching mid-string would let a repo or model name containing a +// quant-shaped substring be reported as the quantization. +var quantizationToken = regexp.MustCompile(`^(?:p?i?q[0-9]+(?:_[0-9a-z]+)*|f(?:p)?(?:8|16|32)|bf16|(?:nv|mx)fp[0-9]+|int[0-9]+|[0-9]+bit)$`) + +// weightExtensions are stripped before a filename is split into segments, so a +// model whose quantization is the last thing in its name (`...-Q8_0.gguf`) is +// not read as the segment `gguf`. +var weightExtensions = []string{".gguf", ".safetensors", ".bin", ".pt", ".pth", ".onnx"} + +// quantizationFromFilename reports the weight format named in a model +// filename, uppercased, or "" when the name does not declare one. +// +// Segments are scanned from the END because that is where the qualifier sits by +// near-universal convention (`Ternary-Bonsai-27B-Q2_g64.gguf`), and because a +// repo path may itself carry a quant-shaped segment that describes the +// repository rather than this file. +// +// Only `-` and `.` split segments; `_` deliberately does not, because it is the +// separator INSIDE a quant token (`Q4_K_M`) and splitting on it would report +// `Q4` for a build that is not Q4. +// +// Two passes, in decreasing order of confidence. A whole segment naming a +// format is the unambiguous case and always wins. Only when no segment does is +// the `_`-delimited tail of a segment considered, which catches the authoring +// style that runs the format into the model name (`gemma-4-E2B_q4_0-it.gguf`, +// a real and populous gallery family). Running the loose pass second rather +// than inline keeps a precise match from ever losing to a fuzzy one further +// right in the name. +// +// The result is uppercased so one gallery does not show `Q2_g64` next to +// `Q2_G64` for what is the same format spelled two ways by two authors. +func quantizationFromFilename(filename string) string { + if filename == "" { + return "" + } + + base := path.Base(strings.TrimSpace(filename)) + lower := strings.ToLower(base) + for _, ext := range weightExtensions { + if strings.HasSuffix(lower, ext) { + base = base[:len(base)-len(ext)] + break + } + } + + segments := strings.FieldsFunc(base, func(r rune) bool { return r == '-' || r == '.' }) + + for i := len(segments) - 1; i >= 0; i-- { + if quantizationToken.MatchString(strings.ToLower(segments[i])) { + return strings.ToUpper(segments[i]) + } + } + + for i := len(segments) - 1; i >= 0; i-- { + if tail := quantizationTail(segments[i]); tail != "" { + return tail + } + } + return "" +} + +// quantizationTail reports a weight format run into the tail of a segment, +// uppercased, or "" when there is none. +// +// It walks the `_`-delimited tails from the LONGEST first, so `e2b_q4_0` yields +// `Q4_0` rather than the `0` a shortest-first walk would reach. The full +// segment is not retried here; the caller has already ruled it out. +func quantizationTail(segment string) string { + for i, r := range segment { + if r != '_' { + continue + } + tail := segment[i+1:] + if quantizationToken.MatchString(strings.ToLower(tail)) { + return strings.ToUpper(tail) + } + } + return "" +} + +// quantizationOfEntry reports the weight format a gallery entry installs. +// +// `overrides.parameters.model` is preferred over the file list because it names +// the file the backend is actually pointed at. An entry routinely ships more +// than one weight file (a vision tower alongside the language model, for +// instance), and those companions carry their own, different quantization: the +// Bonsai entries pair a Q2_0 model with a Q8_0 mmproj, so reading the file list +// first would report Q8_0 for a Q2_0 build. +// +// Returning "" is a normal outcome, not a failure. A backend served from a +// directory of safetensors, or an entry whose name simply does not encode a +// format, has no quantization to report and callers must render its absence +// rather than a guess. +func quantizationOfEntry(entry *GalleryModel) string { + if entry == nil { + return "" + } + + if q := quantizationFromFilename(overrideModelParameter(entry)); q != "" { + return q + } + + // The file list is the fallback for entries that install a config_file + // naming no model parameter of their own. First file wins: the language + // model is listed first by convention, and there is nothing better to go on + // once the authoritative pointer is absent. + for _, f := range entry.AdditionalFiles { + if q := quantizationFromFilename(f.Filename); q != "" { + return q + } + } + return "" +} + +// overrideModelParameter digs `overrides.parameters.model` out of an entry. +// +// Overrides are free-form YAML decoded into map[string]any, so every level has +// to be type-asserted; a gallery author who writes a list or a scalar where a +// map belongs gets "" here rather than a panic in the listing handler. +func overrideModelParameter(entry *GalleryModel) string { + parameters, ok := entry.Overrides["parameters"].(map[string]any) + if !ok { + return "" + } + model, ok := parameters["model"].(string) + if !ok { + return "" + } + return model +} diff --git a/core/gallery/variant_quantization_internal_test.go b/core/gallery/variant_quantization_internal_test.go new file mode 100644 index 000000000000..70f5a1b7f55f --- /dev/null +++ b/core/gallery/variant_quantization_internal_test.go @@ -0,0 +1,126 @@ +package gallery + +import ( + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +var _ = Describe("quantizationFromFilename", func() { + DescribeTable("reads the weight format out of a model filename", + func(filename, expected string) { + Expect(quantizationFromFilename(filename)).To(Equal(expected)) + }, + Entry("a plain legacy quant", "Ternary-Bonsai-27B-Q2_0.gguf", "Q2_0"), + Entry("a packed quant", "Ternary-Bonsai-27B-PQ2_0.gguf", "PQ2_0"), + // The pair the whole feature exists for: two builds of one model whose + // names differ only here, and whose sizes are close enough that a user + // cannot tell them apart from the size column. + Entry("a group-size qualified quant", "Ternary-Bonsai-27B-Q2_g64.gguf", "Q2_G64"), + Entry("a multi-qualifier quant", "Ternary-Bonsai-27B-Q2_0_g128.gguf", "Q2_0_G128"), + Entry("a k-quant", "Qwen3-8B-Q4_K_M.gguf", "Q4_K_M"), + Entry("an i-quant", "Qwen3-8B-IQ2_XXS.gguf", "IQ2_XXS"), + Entry("a float format", "Qwen3-8B-f16.gguf", "F16"), + Entry("bfloat16", "Qwen3-8B-bf16.safetensors", "BF16"), + Entry("an MLX bit count", "Qwen3-8B-4bit.safetensors", "4BIT"), + // A real gallery shape: the format sits mid-name with serving-feature + // and repo-suffix segments after it, so the scan cannot simply take + // the last segment. + Entry("a vendor 4-bit float behind later segments", "Qwen3.6-27B-NVFP4-MTP-GGUF.gguf", "NVFP4"), + Entry("the other vendor 4-bit float", "Qwen3-8B-MXFP4.gguf", "MXFP4"), + // The gemma QAT authoring style: the format is run into the model name + // with an underscore rather than set off by a dash. + Entry("a format run into the model name", "gemma-4-E2B_q4_0-it.gguf", "Q4_0"), + Entry("a format run in at the end", "model_q8_0.gguf", "Q8_0"), + Entry("an integer width", "Qwen3-8B-int8.safetensors", "INT8"), + Entry("a dot-separated qualifier", "qwen3-8b.Q8_0.gguf", "Q8_0"), + // A repo path is routinely prefixed to the filename in the gallery, and + // its own segments must not be mistaken for this file's format. + Entry("a path prefix", "bonsai/models/Q4-repo/Model-Q8_0.gguf", "Q8_0"), + ) + + DescribeTable("reports nothing when the name declares no format", + func(filename string) { + Expect(quantizationFromFilename(filename)).To(BeEmpty()) + }, + Entry("an empty name", ""), + // The degrade case the UI has to render: a backend served from a + // directory of weights names no format anywhere. + Entry("a bare model name", "Qwen3-8B.safetensors"), + Entry("a directory", "models/qwen3-8b/"), + // A parameter count is quant-shaped to a careless matcher: it is a + // digit run next to a letter, and reporting "27B" as a quantization + // would be worse than reporting nothing. + Entry("a parameter count", "Ternary-Bonsai-27B.gguf"), + Entry("an extension alone", "model.gguf"), + ) + + It("prefers a whole segment over a tail further right in the name", func() { + // The two passes exist for this: a precise, dash-delimited format must + // never lose to a looser mid-segment match that happens to sit later. + Expect(quantizationFromFilename("Model-Q4_K_M-repo_8bit.gguf")).To(Equal("Q4_K_M")) + }) + + It("takes the longest tail, not the shortest", func() { + // A shortest-first walk over `e2b_q4_0` reaches `0` before `q4_0`, and + // `0` is not a weight format. + Expect(quantizationFromFilename("gemma_q4_0.gguf")).To(Equal("Q4_0")) + }) + + It("does not split on the separator inside a quant token", func() { + // Splitting on `_` as well as `-` would truncate every k-quant to its + // family and report a Q4_K_M build as plain "Q4", which names a + // different format that the entry does not ship. + Expect(quantizationFromFilename("Model-Q4_K_S.gguf")).To(Equal("Q4_K_S")) + }) +}) + +var _ = Describe("quantizationOfEntry", func() { + It("prefers the served model parameter over the file list", func() { + // The Bonsai shape: a Q2_0 language model shipped alongside a Q8_0 + // vision tower. Reading the file list first reports the mmproj's + // format, which describes a companion the user is not choosing. + entry := &GalleryModel{ + Overrides: map[string]any{ + "parameters": map[string]any{"model": "bonsai/models/Bonsai-27B-Q2_0.gguf"}, + }, + } + entry.AdditionalFiles = []File{ + {Filename: "bonsai/mmproj/Bonsai-27B-mmproj-Q8_0.gguf"}, + {Filename: "bonsai/models/Bonsai-27B-Q2_0.gguf"}, + } + + Expect(quantizationOfEntry(entry)).To(Equal("Q2_0")) + }) + + It("falls back to the file list when no model parameter is set", func() { + entry := &GalleryModel{} + entry.AdditionalFiles = []File{{Filename: "models/Qwen3-8B-Q6_K.gguf"}} + + Expect(quantizationOfEntry(entry)).To(Equal("Q6_K")) + }) + + It("reports nothing for a nil entry", func() { + Expect(quantizationOfEntry(nil)).To(BeEmpty()) + }) + + It("reports nothing when the entry ships no recognisable format", func() { + entry := &GalleryModel{} + entry.AdditionalFiles = []File{{Filename: "models/Qwen3-8B/model.safetensors"}} + + Expect(quantizationOfEntry(entry)).To(BeEmpty()) + }) + + DescribeTable("survives overrides that are not shaped like a parameter map", + func(overrides map[string]any) { + entry := &GalleryModel{Overrides: overrides} + // A gallery author's typo must degrade to "unknown format", never + // panic inside the listing handler. + Expect(quantizationOfEntry(entry)).To(BeEmpty()) + }, + Entry("no overrides at all", nil), + Entry("parameters is a scalar", map[string]any{"parameters": "Q8_0"}), + Entry("parameters is a list", map[string]any{"parameters": []any{"model"}}), + Entry("model is not a string", map[string]any{"parameters": map[string]any{"model": 42}}), + Entry("model is absent", map[string]any{"parameters": map[string]any{"context_size": 8192}}), + ) +}) diff --git a/core/gallery/variants_install_test.go b/core/gallery/variants_install_test.go new file mode 100644 index 000000000000..f677d056170c --- /dev/null +++ b/core/gallery/variants_install_test.go @@ -0,0 +1,757 @@ +package gallery_test + +import ( + "context" + "fmt" + "os" + "path/filepath" + "runtime" + + "dario.cat/mergo" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "gopkg.in/yaml.v3" + + "github.com/mudler/LocalAI/core/config" + "github.com/mudler/LocalAI/core/gallery" + "github.com/mudler/LocalAI/pkg/system" +) + +var _ = Describe("GalleryModel variant declarations", func() { + It("declares no variants when the key is absent", func() { + m := gallery.GalleryModel{} + m.Name = "plain" + Expect(m.HasVariants()).To(BeFalse()) + }) + + It("declares variants when the key is present", func() { + m := gallery.GalleryModel{Variants: []gallery.Variant{{Model: "x"}}} + Expect(m.HasVariants()).To(BeTrue()) + }) + + It("parses a variant list", func() { + var m gallery.GalleryModel + err := yaml.Unmarshal([]byte(` +name: qwen3.6-27b +url: "github:example/repo/qwen3.6-27b.yaml@master" +variants: + - model: qwen3.6-27b-mlx-8bit + - model: qwen3.6-27b-gguf-q8 +`), &m) + Expect(err).ToNot(HaveOccurred()) + Expect(m.Name).To(Equal("qwen3.6-27b")) + Expect(m.URL).To(Equal("github:example/repo/qwen3.6-27b.yaml@master")) + Expect(m.HasVariants()).To(BeTrue()) + Expect(m.Variants).To(HaveLen(2)) + // A variant is nothing but a name, which is the whole of the authoring + // surface. + Expect(m.Variants[0].Model).To(Equal("qwen3.6-27b-mlx-8bit")) + Expect(m.Variants[1].Model).To(Equal("qwen3.6-27b-gguf-q8")) + }) +}) + +var _ = Describe("ResolveVariant", func() { + gib := func(n uint64) uint64 { return n * 1024 * 1024 * 1024 } + + // runsEverything keeps these specs about resolution rather than about the + // hardware of whatever machine runs them. + runsEverything := func(string) bool { return true } + + newModel := func(name, url, description, icon string) *gallery.GalleryModel { + m := &gallery.GalleryModel{} + m.Name = name + m.URL = url + m.Description = description + m.Icon = icon + return m + } + + var models []*gallery.GalleryModel + var base *gallery.GalleryModel + + BeforeEach(func() { + upgrade := newModel("qwen3-8b-vllm-awq", "file://vllm.yaml", "AWQ variant", "vllm.png") + upgrade.Backend = "vllm" + // The base is an ordinary, complete entry that happens to list an + // alternative build of itself, which is the whole point of the design. + base = newModel("qwen3-8b-gguf-q4", "file://gguf.yaml", "Qwen3 8B Q4", "qwen.png") + base.Backend = "llama-cpp" + base.Tags = []string{"llm"} + base.Variants = []gallery.Variant{{Model: "qwen3-8b-vllm-awq"}} + models = []*gallery.GalleryModel{upgrade, base} + }) + + // The AWQ build measures 20GiB, so the specs below vary only the host and + // read off which payload resolution lands on. The probe is injected rather + // than performed, so nothing here reaches the network. + env := func(memory uint64) gallery.ResolveEnv { + return gallery.ResolveEnv{ + AvailableMemory: memory, + BackendCompatible: runsEverything, + ProbeMemory: func(target *gallery.GalleryModel) uint64 { + if target.Name == "qwen3-8b-vllm-awq" { + return 20 * 1024 * 1024 * 1024 + } + return 0 + }, + } + } + + It("installs a fitting variant's payload under the entry's name", func() { + resolved, variant, err := gallery.ResolveVariant(models, base, env(gib(24)), "") + Expect(err).ToNot(HaveOccurred()) + Expect(variant.Model).To(Equal("qwen3-8b-vllm-awq")) + Expect(resolved.Name).To(Equal("qwen3-8b-gguf-q4")) + Expect(resolved.URL).To(Equal("file://vllm.yaml")) + }) + + It("carries the referenced entry's backend into the compatibility check", func() { + // The variant itself says nothing about hardware, so the backend can + // only come from the entry it names. Rejecting exactly that backend must + // therefore be enough to rule the variant out. + resolved, variant, err := gallery.ResolveVariant(models, base, gallery.ResolveEnv{ + AvailableMemory: gib(64), + BackendCompatible: func(backend string) bool { return backend != "vllm" }, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(variant.Model).To(Equal("qwen3-8b-gguf-q4")) + Expect(resolved.URL).To(Equal("file://gguf.yaml")) + }) + + It("carries the referenced entry's tags into the serving feature ranking", func() { + // The tags that rank a variant are the REFERENCED entry's, not the + // declaring entry's: the feature belongs to the build that would be + // installed. Only the GGUF build here is tagged with a feature, and it + // is deliberately the smaller of the two, so it can only win if the + // lookup read a tag off the entry rather than off the variant name or + // off the parent. + speculative := gallery.GalleryModel{} + speculative.Name = "qwen3-8b-gguf-turbo" + speculative.URL = "file://turbo.yaml" + speculative.Backend = "llama-cpp" + speculative.Tags = []string{"llm", "mtp"} + + parent := gallery.GalleryModel{} + parent.Name = "qwen3-8b-gguf-q4" + parent.URL = "file://gguf.yaml" + parent.Backend = "llama-cpp" + parent.Variants = []gallery.Variant{{Model: "qwen3-8b-vllm-awq"}, {Model: "qwen3-8b-gguf-turbo"}} + + catalog := []*gallery.GalleryModel{models[0], &speculative, &parent} + + resolved, variant, err := gallery.ResolveVariant(catalog, &parent, gallery.ResolveEnv{ + AvailableMemory: gib(64), + BackendCompatible: runsEverything, + // Only llama-cpp is ranked, so the vLLM build cannot win on engine + // and the contest comes down to the feature axis. + EnginePreference: []string{"llama-cpp"}, + ServingFeaturePreference: []string{"dflash", "mtp"}, + ProbeMemory: func(target *gallery.GalleryModel) uint64 { + if target.Name == "qwen3-8b-gguf-turbo" { + return gib(10) + } + return gib(20) + }, + }, "") + Expect(err).ToNot(HaveOccurred()) + Expect(variant.Model).To(Equal("qwen3-8b-gguf-turbo")) + Expect(resolved.URL).To(Equal("file://turbo.yaml")) + }) + + It("falls back to the entry's own payload when no variant fits", func() { + resolved, variant, err := gallery.ResolveVariant(models, base, env(gib(8)), "") + Expect(err).ToNot(HaveOccurred()) + Expect(variant.Model).To(Equal("qwen3-8b-gguf-q4")) + Expect(resolved.URL).To(Equal("file://gguf.yaml")) + }) + + It("installs the entry on a host with no memory budget at all", func() { + // The base carries no floor of its own precisely so this can never + // refuse; an unreadable host reports 0 and must still get the entry. + resolved, variant, err := gallery.ResolveVariant(models, base, env(0), "") + Expect(err).ToNot(HaveOccurred()) + Expect(variant.Model).To(Equal("qwen3-8b-gguf-q4")) + Expect(resolved.URL).To(Equal("file://gguf.yaml")) + }) + + It("strips the variant list from the resolved entry", func() { + // A resolved entry is a concrete install target. Leaving the list on it + // would let a second selection pass fire on an already-resolved entry. + resolved, _, err := gallery.ResolveVariant(models, base, env(gib(8)), "") + Expect(err).ToNot(HaveOccurred()) + Expect(resolved.HasVariants()).To(BeFalse()) + }) + + Describe("the live size probe", func() { + // The probe is injected rather than performed, so these specs pin an + // exact size, or an exact failure, without reaching the network. + probing := func(memory uint64, sizes map[string]uint64) gallery.ResolveEnv { + e := env(memory) + e.ProbeMemory = func(target *gallery.GalleryModel) uint64 { return sizes[target.Name] } + return e + } + + It("selects a variant the probe shows to fit", func() { + resolved, variant, err := gallery.ResolveVariant(models, base, + probing(gib(24), map[string]uint64{"qwen3-8b-vllm-awq": gib(20)}), "") + Expect(err).ToNot(HaveOccurred()) + Expect(variant.Model).To(Equal("qwen3-8b-vllm-awq")) + Expect(resolved.URL).To(Equal("file://vllm.yaml")) + }) + + It("rejects a variant the probe shows to be too large", func() { + // Same variant, same host, only the probed size differs, so nothing + // but the probe can account for the different answer. + resolved, variant, err := gallery.ResolveVariant(models, base, + probing(gib(24), map[string]uint64{"qwen3-8b-vllm-awq": gib(80)}), "") + Expect(err).ToNot(HaveOccurred()) + Expect(variant.Model).To(Equal("qwen3-8b-gguf-q4")) + Expect(resolved.URL).To(Equal("file://gguf.yaml")) + }) + + It("completes the install on the entry's own payload when no probe answers", func() { + // A failed probe reports 0. That is an unknown, so nothing is filtered + // out and above all the resolution does not fail: a network hiccup + // must never be able to break an install. It must not be able to + // change what gets installed either, so an unmeasurable variant does + // not displace the payload the entry itself ships. + resolved, variant, err := gallery.ResolveVariant(models, base, + probing(gib(24), map[string]uint64{}), "") + Expect(err).ToNot(HaveOccurred()) + Expect(variant.Model).To(Equal("qwen3-8b-gguf-q4")) + Expect(resolved.URL).To(Equal("file://gguf.yaml")) + }) + + It("still installs the base when every probe fails and nothing else survives", func() { + // The one variant this host could otherwise take is ruled out by its + // backend and no probe answers, so selection has to terminate on the + // base rather than on an error. + e := probing(gib(24), map[string]uint64{}) + e.BackendCompatible = func(backend string) bool { return backend != "vllm" } + + resolved, variant, err := gallery.ResolveVariant(models, base, e, "") + Expect(err).ToNot(HaveOccurred()) + Expect(variant.Model).To(Equal("qwen3-8b-gguf-q4")) + Expect(resolved.URL).To(Equal("file://gguf.yaml")) + }) + }) + + It("presents the entry's metadata, not the variant's", func() { + resolved, _, err := gallery.ResolveVariant(models, base, env(gib(24)), "") + Expect(err).ToNot(HaveOccurred()) + Expect(resolved.Description).To(Equal("Qwen3 8B Q4")) + Expect(resolved.Icon).To(Equal("qwen.png")) + Expect(resolved.Tags).To(ConsistOf("llm")) + }) + + It("honors a pin naming the entry itself", func() { + // The entry is the last resort of its own variant list, so its own name + // has to be a usable pin: it is how an operator declines an upgrade + // their hardware would otherwise take. + resolved, variant, err := gallery.ResolveVariant(models, base, env(gib(24)), "qwen3-8b-gguf-q4") + Expect(err).ToNot(HaveOccurred()) + Expect(variant.Model).To(Equal("qwen3-8b-gguf-q4")) + Expect(resolved.Name).To(Equal("qwen3-8b-gguf-q4")) + Expect(resolved.URL).To(Equal("file://gguf.yaml")) + }) + + It("honors a pin the hardware does not satisfy", func() { + resolved, variant, err := gallery.ResolveVariant(models, base, env(gib(2)), "qwen3-8b-vllm-awq") + Expect(err).ToNot(HaveOccurred()) + Expect(variant.Model).To(Equal("qwen3-8b-vllm-awq")) + Expect(resolved.URL).To(Equal("file://vllm.yaml")) + }) + + It("detaches the resolved entry from the gallery's own maps and slices", func() { + models[0].Overrides = map[string]any{"f16": true} + models[0].AdditionalFiles = []gallery.File{{Filename: "a.bin"}} + + resolved, _, err := gallery.ResolveVariant(models, base, env(gib(24)), "") + Expect(err).ToNot(HaveOccurred()) + + // The install path merges the caller's request overrides into this map in + // place, so aliasing the gallery's map would write one caller's request + // into the shared catalog and leak it into every later install. + resolved.Overrides["f16"] = false + resolved.Overrides["threads"] = 4 + Expect(models[0].Overrides).To(HaveKeyWithValue("f16", true)) + Expect(models[0].Overrides).ToNot(HaveKey("threads")) + + resolved.AdditionalFiles[0].Filename = "mutated.bin" + Expect(models[0].AdditionalFiles[0].Filename).To(Equal("a.bin")) + + resolved.Tags = append(resolved.Tags, "extra") + Expect(base.Tags).To(ConsistOf("llm")) + }) + + It("detaches the resolved entry even when it resolves to the entry itself", func() { + // Resolving to the base returns a copy of the very entry the gallery + // holds, which is the case most likely to alias it. + base.Overrides = map[string]any{"parameters": map[string]any{"model": "q4.gguf"}} + + resolved, _, err := gallery.ResolveVariant(models, base, env(gib(8)), "") + Expect(err).ToNot(HaveOccurred()) + + resolved.Overrides["parameters"].(map[string]any)["model"] = "callers-choice.gguf" + Expect(base.Overrides["parameters"]).To(HaveKeyWithValue("model", "q4.gguf")) + }) + + It("detaches nested override maps from the gallery's own entry", func() { + models[0].Overrides = map[string]any{ + "parameters": map[string]any{"model": "real-variant.gguf"}, + "stopwords": []any{""}, + } + + resolved, _, err := gallery.ResolveVariant(models, base, env(gib(24)), "") + Expect(err).ToNot(HaveOccurred()) + + // Cloning only the top level would leave this inner map shared with the + // gallery entry, so writing through the resolved copy would rewrite the + // catalog's own payload. + resolved.Overrides["parameters"].(map[string]any)["model"] = "callers-choice.gguf" + resolved.Overrides["stopwords"].([]any)[0] = "<|im_end|>" + + Expect(models[0].Overrides["parameters"]).To(HaveKeyWithValue("model", "real-variant.gguf")) + Expect(models[0].Overrides["stopwords"]).To(Equal([]any{""})) + }) + + It("does not write the caller's overrides back into the gallery entry", func() { + models[0].Overrides = map[string]any{"parameters": map[string]any{"model": "real-variant.gguf"}} + + resolved, _, err := gallery.ResolveVariant(models, base, env(gib(24)), "") + Expect(err).ToNot(HaveOccurred()) + + // This is exactly what the install path does with the caller's request + // overrides, and mergo recurses into nested maps and overwrites them in + // place. Asserting against the in-memory catalog is the only way to + // observe the leak: re-reading the gallery from disk re-unmarshals fresh + // maps and would pass whether or not the resolved entry was detached. + requestOverrides := map[string]any{"parameters": map[string]any{"model": "callers-choice.gguf"}} + Expect(mergo.Merge(&resolved.Overrides, requestOverrides, mergo.WithOverride)).To(Succeed()) + + Expect(resolved.Overrides["parameters"]).To(HaveKeyWithValue("model", "callers-choice.gguf")) + Expect(models[0].Overrides["parameters"]).To(HaveKeyWithValue("model", "real-variant.gguf")) + }) + + It("errors when a variant references a missing entry", func() { + base.Variants = []gallery.Variant{{Model: "does-not-exist"}} + _, _, err := gallery.ResolveVariant(models, base, env(gib(8)), "") + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("does-not-exist")) + }) + + It("refuses a variant that declares variants of its own", func() { + nested := newModel("nested", "file://nested.yaml", "", "") + nested.Variants = []gallery.Variant{{Model: "qwen3-8b-vllm-awq"}} + models = append(models, nested) + base.Variants = []gallery.Variant{{Model: "nested"}} + _, _, err := gallery.ResolveVariant(models, base, env(gib(8)), "") + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("nested")) + }) + + It("surfaces a bad pin", func() { + _, _, err := gallery.ResolveVariant(models, base, env(gib(24)), "nope") + Expect(err).To(MatchError(gallery.ErrPinNotFound)) + }) +}) + +var _ = Describe("InstallModelFromGallery with variant entries", func() { + var tempdir string + var galleries []config.Gallery + var systemState *system.SystemState + // galleryRevision keeps every gallery this suite writes distinct from every + // other, so the process-wide gallery cache cannot serve one spec's catalog + // to another. + galleryRevision := 0 + + // Every entry is described with an inline config_file rather than a URL so + // the whole install runs off the local filesystem with no network access. + // + // Each call writes a fresh path and a fresh gallery name because the gallery + // listing is cached on the name and URL pair. A spec that edits its gallery + // mid-flight would otherwise keep reading the version it started with, and + // would silently prove nothing. + newGallery := func(entries ...gallery.GalleryModel) { + out, err := yaml.Marshal(entries) + Expect(err).ToNot(HaveOccurred()) + name := fmt.Sprintf("test-%d", galleryRevision) + galleryRevision++ + galleryPath := filepath.Join(tempdir, name+".yaml") + Expect(os.WriteFile(galleryPath, out, 0600)).To(Succeed()) + + galleries = []config.Gallery{{Name: name, URL: "file://" + galleryPath}} + } + + entry := func(name, backend string) gallery.GalleryModel { + m := gallery.GalleryModel{ConfigFile: map[string]any{"backend": backend}} + m.Name = name + m.Description = "entry " + name + return m + } + + // sizedEntry declares its footprint through the entry's own size:, which is + // the only way an author influences a variant's measured size. The estimator + // prefers a declared size over its own guesswork and parses it locally, so + // these specs pin an exact figure without touching the network. + sizedEntry := func(name, backend, size string) gallery.GalleryModel { + m := entry(name, backend) + m.Size = size + return m + } + + // urlEntry describes an entry through a url rather than an inline + // config_file. The distinction matters for the recorded name: the url branch + // reads a name out of the fetched config, and that name is the referenced + // entry's own, so only the entry-name overlay can keep the record under the + // name the user asked for. The inline config_file branch seeds the name from + // the already-renamed resolved entry and so cannot observe the overlay. + urlEntry := func(name, backend, size string) gallery.GalleryModel { + payload := gallery.ModelConfig{ + Name: name, + Description: "entry " + name, + ConfigFile: "backend: " + backend + "\n", + } + out, err := yaml.Marshal(payload) + Expect(err).ToNot(HaveOccurred()) + payloadPath := filepath.Join(tempdir, "payload-"+name+".yaml") + Expect(os.WriteFile(payloadPath, out, 0600)).To(Succeed()) + + m := gallery.GalleryModel{} + m.Name = name + m.Description = "entry " + name + m.URL = "file://" + payloadPath + m.Size = size + return m + } + + // withVariants attaches alternative builds to an otherwise ordinary entry. + // Where a spec needs a definite size it declares it on the referenced entry + // with sizedEntry, in absolute terms rather than relative to the host, so + // these specs assert on selection rather than on whatever memory the machine + // running them happens to have. + withVariants := func(m gallery.GalleryModel, variants ...gallery.Variant) gallery.GalleryModel { + m.Variants = variants + return m + } + + install := func(name string, req gallery.GalleryModel, options ...gallery.InstallOption) error { + return gallery.InstallModelFromGallery( + context.TODO(), galleries, []config.Gallery{}, systemState, nil, + name, req, func(string, string, string, float64) {}, false, false, false, options...) + } + + installedBackend := func(name string) string { + dat, err := os.ReadFile(filepath.Join(tempdir, name+".yaml")) + Expect(err).ToNot(HaveOccurred()) + content := map[string]any{} + Expect(yaml.Unmarshal(dat, &content)).To(Succeed()) + return content["backend"].(string) + } + + BeforeEach(func() { + var err error + tempdir, err = os.MkdirTemp("", "variant-install") + Expect(err).ToNot(HaveOccurred()) + DeferCleanup(func() { Expect(os.RemoveAll(tempdir)).To(Succeed()) }) + + systemState, err = system.GetSystemState(system.WithModelPath(tempdir)) + Expect(err).ToNot(HaveOccurred()) + }) + + It("installs the entry's own payload when no variant fits the host", func() { + newGallery( + withVariants(entry("qwen3-8b-q4", "base-backend"), + gallery.Variant{Model: "qwen3-8b-q8"}), + sizedEntry("qwen3-8b-q8", "upgrade-backend", "10000GiB"), + ) + + // No machine clears 10000GiB, so this asserts the base is the last + // resort and that missing every variant is not an error. + Expect(install("qwen3-8b-q4", gallery.GalleryModel{})).To(Succeed()) + Expect(installedBackend("qwen3-8b-q4")).To(Equal("base-backend")) + }) + + It("installs a fitting variant's payload under the entry's own name", func() { + newGallery( + withVariants(entry("qwen3-8b-q4", "base-backend"), + gallery.Variant{Model: "qwen3-8b-q8"}), + sizedEntry("qwen3-8b-q8", "upgrade-backend", "16MiB"), + ) + + Expect(install("qwen3-8b-q4", gallery.GalleryModel{})).To(Succeed()) + Expect(installedBackend("qwen3-8b-q4")).To(Equal("upgrade-backend")) + _, err := os.Stat(filepath.Join(tempdir, "qwen3-8b-q8.yaml")) + Expect(os.IsNotExist(err)).To(BeTrue(), "the variant must not be installed under its own name") + }) + + It("completes the install when a variant's size cannot be determined", func() { + // This runs the REAL probe against an entry that declares no size and no + // weight files, which is exactly the shape a probe cannot answer for. + // It must yield an unknown, not an error: the variant survives the + // filter and the install completes, on the variant or on the base, but + // never on a failure. Nothing here touches the network. + newGallery( + withVariants(entry("qwen3-8b-q4", "base-backend"), + gallery.Variant{Model: "qwen3-8b-q8"}), + entry("qwen3-8b-q8", "upgrade-backend"), + ) + + Expect(install("qwen3-8b-q4", gallery.GalleryModel{})).To(Succeed()) + Expect(installedBackend("qwen3-8b-q4")).To(BeElementOf("base-backend", "upgrade-backend")) + }) + + It("installs the largest fitting variant, not the first authored", func() { + // Authored smallest-first with all three within reach, so first-match + // would take the small one. + newGallery( + withVariants(entry("qwen3-8b-q4", "base-backend"), + gallery.Variant{Model: "qwen3-8b-small"}, + gallery.Variant{Model: "qwen3-8b-large"}, + ), + sizedEntry("qwen3-8b-small", "small-backend", "16MiB"), + sizedEntry("qwen3-8b-large", "large-backend", "256MiB"), + ) + + Expect(install("qwen3-8b-q4", gallery.GalleryModel{})).To(Succeed()) + Expect(installedBackend("qwen3-8b-q4")).To(Equal("large-backend")) + }) + + It("never installs a variant whose backend cannot run on this host", func() { + if runtime.GOOS == "darwin" { + Skip("mlx is a compatible backend on darwin, so it proves nothing here") + } + // The mlx entry is the only variant and fits trivially, so the real + // SystemState.IsBackendCompatible rejecting mlx on a non-darwin host is + // the only thing that can send this to the base. + newGallery( + withVariants(entry("qwen3-8b-q4", "base-backend"), + gallery.Variant{Model: "qwen3-8b-mlx"}), + entry("qwen3-8b-mlx", "mlx"), + ) + + Expect(install("qwen3-8b-q4", gallery.GalleryModel{})).To(Succeed()) + Expect(installedBackend("qwen3-8b-q4")).To(Equal("base-backend")) + }) + + It("round-trips the resolution record to disk under the entry's name", func() { + // The variant is described by url so its payload carries its own name. + // Without the entry-name overlay the record persists as "qwen3-8b-q8", + // and the stable name the entry exists to provide is lost the moment + // anything reads the record back. + // The declared sizes make the upgrade the largest build that fits, so + // resolution lands on the variant and the record it writes is the one + // under test here. + newGallery( + withVariants(urlEntry("qwen3-8b-q4", "base-backend", "16MiB"), + gallery.Variant{Model: "qwen3-8b-q8"}), + urlEntry("qwen3-8b-q8", "upgrade-backend", "256MiB"), + ) + + Expect(install("qwen3-8b-q4", gallery.GalleryModel{})).To(Succeed()) + + record, err := gallery.GetLocalModelConfiguration(tempdir, "qwen3-8b-q4") + Expect(err).ToNot(HaveOccurred()) + Expect(record.EntryName).To(Equal("qwen3-8b-q4")) + Expect(record.ResolvedVariant).To(Equal("qwen3-8b-q8")) + Expect(record.PinnedVariant).To(BeEmpty()) + Expect(record.Name).To(Equal("qwen3-8b-q4")) + Expect(record.Description).To(Equal("entry qwen3-8b-q4")) + Expect(installedBackend("qwen3-8b-q4")).To(Equal("upgrade-backend")) + }) + + It("refuses a variant name the entry does not declare, naming what was asked for", func() { + newGallery( + withVariants(entry("qwen3-8b-q4", "base-backend"), + gallery.Variant{Model: "qwen3-8b-q8"}), + entry("qwen3-8b-q8", "upgrade-backend"), + ) + + err := install("qwen3-8b-q4", gallery.GalleryModel{}, gallery.WithVariant("qwen3-8b-q6")) + + // Auto-selecting instead would report success while silently + // installing something other than what the caller asked for, and a + // typo'd variant name would be indistinguishable from a working one. + Expect(err).To(MatchError(gallery.ErrPinNotFound)) + Expect(err.Error()).To(ContainSubstring("qwen3-8b-q6")) + Expect(filepath.Join(tempdir, "qwen3-8b-q4.yaml")).ToNot(BeAnExistingFile()) + }) + + It("refuses a variant name when the entry declares no variants at all", func() { + // The entry is installable and selection never runs for it, so without + // an explicit refusal the requested variant would be dropped on the + // floor and the install would look like it honored the choice. + newGallery(entry("qwen3-8b-q4", "base-backend")) + + err := install("qwen3-8b-q4", gallery.GalleryModel{}, gallery.WithVariant("qwen3-8b-q8")) + + Expect(err).To(MatchError(gallery.ErrPinNotFound)) + Expect(err.Error()).To(ContainSubstring("qwen3-8b-q8")) + Expect(filepath.Join(tempdir, "qwen3-8b-q4.yaml")).ToNot(BeAnExistingFile()) + }) + + It("records a pin and honors it on a plain reinstall", func() { + // The upgrade declares a size and the base does not, so auto-selection + // genuinely prefers the upgrade here. Without that the second install + // below would land on the base whether or not the pin was recalled, and + // the spec would prove nothing. + newGallery( + withVariants(entry("qwen3-8b-q4", "base-backend"), + gallery.Variant{Model: "qwen3-8b-q8"}), + sizedEntry("qwen3-8b-q8", "upgrade-backend", "16MiB"), + ) + + Expect(install("qwen3-8b-q4", gallery.GalleryModel{}, gallery.WithVariant("qwen3-8b-q4"))).To(Succeed()) + + record, err := gallery.GetLocalModelConfiguration(tempdir, "qwen3-8b-q4") + Expect(err).ToNot(HaveOccurred()) + Expect(record.PinnedVariant).To(Equal("qwen3-8b-q4")) + Expect(record.ResolvedVariant).To(Equal("qwen3-8b-q4")) + Expect(installedBackend("qwen3-8b-q4")).To(Equal("base-backend")) + + // No WithVariant this time: hardware selection would take the upgrade, + // so only the recalled pin can keep this on the base payload. + Expect(install("qwen3-8b-q4", gallery.GalleryModel{})).To(Succeed()) + Expect(installedBackend("qwen3-8b-q4")).To(Equal("base-backend")) + + record, err = gallery.GetLocalModelConfiguration(tempdir, "qwen3-8b-q4") + Expect(err).ToNot(HaveOccurred()) + Expect(record.PinnedVariant).To(Equal("qwen3-8b-q4")) + }) + + It("honors a pin recorded under a custom install name", func() { + // Sized as above so auto-selection would take the upgrade, which is what + // makes the recall observable in the second install. + newGallery( + withVariants(entry("qwen3-8b-q4", "base-backend"), + gallery.Variant{Model: "qwen3-8b-q8"}), + sizedEntry("qwen3-8b-q8", "upgrade-backend", "16MiB"), + ) + + req := gallery.GalleryModel{} + req.Name = "prod-llm" + + Expect(install("qwen3-8b-q4", req, gallery.WithVariant("qwen3-8b-q4"))).To(Succeed()) + + // The pin is written under the installed name, so the recall must read it + // back under that name too and not under the gallery entry's name. + record, err := gallery.GetLocalModelConfiguration(tempdir, "prod-llm") + Expect(err).ToNot(HaveOccurred()) + Expect(record.PinnedVariant).To(Equal("qwen3-8b-q4")) + Expect(record.EntryName).To(Equal("qwen3-8b-q4")) + Expect(installedBackend("prod-llm")).To(Equal("base-backend")) + + Expect(install("qwen3-8b-q4", req)).To(Succeed()) + Expect(installedBackend("prod-llm")).To(Equal("base-backend")) + }) + + It("re-selects automatically when a recorded pin no longer exists in the gallery", func() { + // A pin is recorded, then the gallery is edited to rename the build it + // named. Without dropping the stale pin every later reinstall and every + // upgrade of this model fails forever, and the only remedy is deleting a + // dotfile the user has never heard of. + newGallery( + withVariants(entry("qwen3-8b-q4", "base-backend"), + gallery.Variant{Model: "qwen3-8b-q8"}), + sizedEntry("qwen3-8b-q8", "upgrade-backend", "16MiB"), + ) + Expect(install("qwen3-8b-q4", gallery.GalleryModel{}, gallery.WithVariant("qwen3-8b-q8"))).To(Succeed()) + + record, err := gallery.GetLocalModelConfiguration(tempdir, "qwen3-8b-q4") + Expect(err).ToNot(HaveOccurred()) + Expect(record.PinnedVariant).To(Equal("qwen3-8b-q8")) + + // The gallery edit: the same build under a new name. + newGallery( + withVariants(entry("qwen3-8b-q4", "base-backend"), + gallery.Variant{Model: "qwen3-8b-q8-v2"}), + sizedEntry("qwen3-8b-q8-v2", "renamed-backend", "16MiB"), + ) + + Expect(install("qwen3-8b-q4", gallery.GalleryModel{})).To(Succeed()) + // Auto-selection ran: the renamed build is the largest that fits. + Expect(installedBackend("qwen3-8b-q4")).To(Equal("renamed-backend")) + + // The stale pin is cleared rather than carried forward, so the next + // install does not have to rediscover that it is unusable. + record, err = gallery.GetLocalModelConfiguration(tempdir, "qwen3-8b-q4") + Expect(err).ToNot(HaveOccurred()) + Expect(record.PinnedVariant).To(BeEmpty()) + Expect(record.ResolvedVariant).To(Equal("qwen3-8b-q8-v2")) + }) + + It("still refuses a variant the caller names on this request, recorded pin or not", func() { + // The caller-supplied pin and the recalled one must fail differently. + // This one is a request that cannot be honored, so reporting success + // for it would make a typo indistinguishable from a working choice. + newGallery( + withVariants(entry("qwen3-8b-q4", "base-backend"), + gallery.Variant{Model: "qwen3-8b-q8"}), + sizedEntry("qwen3-8b-q8", "upgrade-backend", "16MiB"), + ) + Expect(install("qwen3-8b-q4", gallery.GalleryModel{}, gallery.WithVariant("qwen3-8b-q8"))).To(Succeed()) + + err := install("qwen3-8b-q4", gallery.GalleryModel{}, gallery.WithVariant("qwen3-8b-q6")) + Expect(err).To(MatchError(gallery.ErrPinNotFound)) + Expect(err.Error()).To(ContainSubstring("qwen3-8b-q6")) + }) + + It("writes each declared url only once into the persisted gallery file", func() { + base := withVariants(entry("qwen3-8b-q4", "base-backend"), + gallery.Variant{Model: "qwen3-8b-q8"}) + base.URLs = []string{"https://example.invalid/qwen3"} + newGallery(base, entry("qwen3-8b-q8", "upgrade-backend")) + + Expect(install("qwen3-8b-q4", gallery.GalleryModel{})).To(Succeed()) + + record, err := gallery.GetLocalModelConfiguration(tempdir, "qwen3-8b-q4") + Expect(err).ToNot(HaveOccurred()) + Expect(record.URLs).To(ConsistOf("https://example.invalid/qwen3")) + }) +}) + +var _ = Describe("legacy client compatibility", func() { + It("keeps every entry that declares variants installable by clients that ignore them", func() { + data, err := os.ReadFile(filepath.Join("..", "..", "gallery", "index.yaml")) + Expect(err).ToNot(HaveOccurred()) + + // Parse exactly as an older LocalAI release would: non-strictly, with + // no knowledge of the variants key. Such a client installs whatever + // payload the entry carries directly, so the entry must carry one. + var legacy []struct { + Name string `yaml:"name"` + URL string `yaml:"url"` + ConfigFile map[string]any `yaml:"config_file"` + Files []gallery.File `yaml:"files"` + Overrides map[string]any `yaml:"overrides"` + } + Expect(yaml.Unmarshal(data, &legacy)).To(Succeed()) + + var current []gallery.GalleryModel + Expect(yaml.Unmarshal(data, ¤t)).To(Succeed()) + + legacyByName := map[string]int{} + for i, e := range legacy { + legacyByName[e.Name] = i + } + + withVariants := 0 + for _, e := range current { + if !e.HasVariants() { + continue + } + withVariants++ + + i, ok := legacyByName[e.Name] + Expect(ok).To(BeTrue(), "entry %q vanished under a legacy parse", e.Name) + old := legacy[i] + // An entry whose payload lived only in its variants would install to + // nothing on every released LocalAI, which is precisely what making + // the entry itself the last resort exists to prevent. + Expect(old.URL != "" || len(old.ConfigFile) > 0).To(BeTrue(), + "entry %q carries no payload of its own, so older clients would install nothing", e.Name) + } + Expect(withVariants).To(BeNumerically(">", 0), + "expected at least one entry declaring variants in the gallery index") + }) +}) diff --git a/core/gallery/variants_lint_test.go b/core/gallery/variants_lint_test.go new file mode 100644 index 000000000000..c5e0bda3022f --- /dev/null +++ b/core/gallery/variants_lint_test.go @@ -0,0 +1,400 @@ +package gallery_test + +import ( + "fmt" + "os" + "path/filepath" + "strings" + "sync" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "gopkg.in/yaml.v3" + + "github.com/mudler/LocalAI/core/gallery" +) + +// variantViolation is one invariant breach found by a lint helper. Helpers +// return every breach they find instead of stopping at the first, so a single +// run names them all rather than forcing a fix-one-rerun-repeat cycle. +type variantViolation struct { + Entry string + Variant string + Detail string +} + +func (v variantViolation) String() string { + if v.Variant == "" { + return fmt.Sprintf("%s: %s", v.Entry, v.Detail) + } + return fmt.Sprintf("%s -> variant %q: %s", v.Entry, v.Variant, v.Detail) +} + +func formatViolations(violations []variantViolation) string { + lines := make([]string, 0, len(violations)) + for _, v := range violations { + lines = append(lines, " "+v.String()) + } + return "\n" + strings.Join(lines, "\n") +} + +func indexEntriesByName(entries []gallery.GalleryModel) map[string]gallery.GalleryModel { + byName := make(map[string]gallery.GalleryModel, len(entries)) + for _, e := range entries { + byName[e.Name] = e + } + return byName +} + +// checkVariantReferences verifies every variant names an existing entry that +// declares no variants of its own. Selection is a single pass, so a nested +// reference would silently ignore the inner list. +// +// This is the one structural rule left. The rules about authored order, the +// hardware vocabulary and floor relationships stopped describing real hazards +// once selection became a filter plus a ranking: an author writes names, and +// there is no longer anything about hardware they can get wrong. +func checkVariantReferences(entries []gallery.GalleryModel) []variantViolation { + byName := indexEntriesByName(entries) + var violations []variantViolation + for _, e := range entries { + if !e.HasVariants() { + continue + } + for _, v := range e.Variants { + target, ok := byName[v.Model] + if !ok { + violations = append(violations, variantViolation{Entry: e.Name, Variant: v.Model, Detail: "references unknown model"}) + continue + } + if target.HasVariants() { + violations = append(violations, variantViolation{Entry: e.Name, Variant: v.Model, Detail: "references an entry that declares variants of its own; nesting is not allowed"}) + } + } + } + return violations +} + +// checkEntriesInstallSomething verifies every entry would actually put +// something on disk. +// +// It replaces an earlier rule that demanded a url: or a config_file: from every +// variant target. That demand no longer holds: applyModel now treats an entry +// declaring neither as an empty base config, which is precisely what the many +// entries pointing at gallery/virtual.yaml were already getting, minus the +// fetch. Requiring one of the two fields would now reject perfectly good +// authoring. +// +// What survives is the weaker invariant the relaxation left exposed. An entry +// with no base config, no overrides: and no files: names nothing to download +// and nothing to configure, so installing it yields an empty model directory. +// That is an authoring mistake, and it is worth catching in the catalog rather +// than on someone's machine. +// +// The rule covers every entry, not only variant targets, because the hazard has +// nothing to do with variants: it is a half-written stanza, and a parent entry +// can be one just as easily as a target. +// entryInstallsSomething restates applyModel's acceptance rule in the terms an +// author reads: a base config, or a payload to lay over an empty one. +func entryInstallsSomething(e gallery.GalleryModel) bool { + return len(e.URL) > 0 || len(e.ConfigFile) > 0 || len(e.Overrides) > 0 || len(e.AdditionalFiles) > 0 +} + +func checkEntriesInstallSomething(entries []gallery.GalleryModel) []variantViolation { + var violations []variantViolation + for _, e := range entries { + if entryInstallsSomething(e) { + continue + } + violations = append(violations, variantViolation{ + Entry: e.Name, + Detail: fmt.Sprintf("entry %q installs nothing: it declares no url:, no config_file:, no overrides: and no files:, "+ + "so installing it would leave an empty model directory. Give it the payload it is missing. "+ + "Note that urls: (plural) is the informational link list and is not a payload", e.Name), + }) + } + return violations +} + +// loadGalleryIndex parses gallery/index.yaml once for the whole suite. The +// index carries well over a thousand entries, so re-parsing it per spec is +// pure overhead. +var loadGalleryIndex = sync.OnceValues(func() ([]gallery.GalleryModel, error) { + data, err := os.ReadFile(filepath.Join("..", "..", "gallery", "index.yaml")) + if err != nil { + return nil, err + } + var entries []gallery.GalleryModel + if err := yaml.Unmarshal(data, &entries); err != nil { + return nil, err + } + return entries, nil +}) + +func plainEntry(name, url string) gallery.GalleryModel { + e := gallery.GalleryModel{} + e.Name = name + e.URL = url + return e +} + +func entryWithVariants(name, url string, variants ...gallery.Variant) gallery.GalleryModel { + e := gallery.GalleryModel{Variants: variants} + e.Name = name + e.URL = url + return e +} + +// variantFixture builds an entry with variants plus the entries it references. +// Synthetic fixtures keep the invariant logic covered however few entries in +// gallery/index.yaml declare variants; without them every index-driven spec +// below is a no-op that passes while checking nothing. +func variantFixture(base gallery.GalleryModel, referenced ...gallery.GalleryModel) []gallery.GalleryModel { + return append([]gallery.GalleryModel{base}, referenced...) +} + +var _ = Describe("gallery variant lint helpers", func() { + // An entry declares variants solely by carrying the key. GalleryBackend.IsMeta() + // has deliberately different semantics (it requires an EMPTY uri), so a + // well-meaning alignment of the two would make every helper skip every + // entry and pass silently. Assert the distinction directly. + It("treats an entry with variants as such even though it has a url", func() { + Expect(entryWithVariants("base", "u://base", gallery.Variant{Model: "big"}).HasVariants()).To(BeTrue()) + Expect(plainEntry("big", "u://big").HasVariants()).To(BeFalse()) + }) + + It("passes every invariant on a valid entry", func() { + // Authoring is nothing but a list of names, so a bare list of them must + // be entirely valid. + entries := variantFixture( + entryWithVariants("base", "u://base", + gallery.Variant{Model: "big"}, + gallery.Variant{Model: "mid"}, + gallery.Variant{Model: "metal-big"}, + gallery.Variant{Model: "small"}, + ), + plainEntry("big", "u://big"), + plainEntry("mid", "u://mid"), + plainEntry("metal-big", "u://metal-big"), + plainEntry("small", "u://small"), + ) + + Expect(checkVariantReferences(entries)).To(BeEmpty()) + Expect(checkEntriesInstallSomething(entries)).To(BeEmpty()) + }) + + Describe("checkEntriesInstallSomething", func() { + // The shape the rule exists for: a stanza that got as far as a name and + // stopped. + empty := func(name string) gallery.GalleryModel { + e := gallery.GalleryModel{} + e.Name = name + // urls: is the informational HuggingFace link list. It reads like a + // payload and is not one, so a stub carrying only this must still be + // flagged. + e.URLs = []string{"https://huggingface.co/example/" + name} + return e + } + + It("flags an entry with no url, no config_file, no overrides and no files", func() { + entries := variantFixture( + entryWithVariants("base", "u://base", gallery.Variant{Model: "dead"}), + empty("dead"), + ) + + violations := checkEntriesInstallSomething(entries) + Expect(violations).To(HaveLen(1)) + Expect(violations[0].Entry).To(Equal("dead")) + Expect(violations[0].Detail).To(ContainSubstring("installs nothing")) + }) + + It("accepts an entry carrying only overrides, which applyModel installs on an empty base", func() { + target := gallery.GalleryModel{Overrides: map[string]any{"backend": "ds4"}} + target.Name = "overrides-only" + entries := variantFixture( + entryWithVariants("base", "u://base", gallery.Variant{Model: "overrides-only"}), + target, + ) + + Expect(checkEntriesInstallSomething(entries)).To(BeEmpty()) + }) + + It("accepts an entry carrying only files", func() { + target := gallery.GalleryModel{} + target.Name = "files-only" + target.AdditionalFiles = []gallery.File{{Filename: "weights.gguf", URI: "u://weights"}} + entries := variantFixture( + entryWithVariants("base", "u://base", gallery.Variant{Model: "files-only"}), + target, + ) + + Expect(checkEntriesInstallSomething(entries)).To(BeEmpty()) + }) + + It("accepts an entry described by an inline config_file rather than a url", func() { + target := gallery.GalleryModel{ConfigFile: map[string]any{"backend": "llama-cpp"}} + target.Name = "inline" + entries := variantFixture( + entryWithVariants("base", "u://base", gallery.Variant{Model: "inline"}), + target, + ) + + Expect(checkEntriesInstallSomething(entries)).To(BeEmpty()) + }) + + It("reports every breach in one pass rather than stopping at the first", func() { + entries := variantFixture( + entryWithVariants("base", "u://base", + gallery.Variant{Model: "dead-a"}, + gallery.Variant{Model: "dead-b"}, + ), + empty("dead-a"), + empty("dead-b"), + ) + + Expect(checkEntriesInstallSomething(entries)).To(HaveLen(2)) + }) + }) + + Describe("checkVariantReferences", func() { + It("flags a variant naming an entry that does not exist", func() { + entries := variantFixture( + entryWithVariants("base", "u://base", gallery.Variant{Model: "ghost"}), + plainEntry("a", "u://a"), + ) + + violations := checkVariantReferences(entries) + Expect(violations).To(HaveLen(1)) + Expect(violations[0].Variant).To(Equal("ghost")) + Expect(violations[0].Detail).To(ContainSubstring("unknown model")) + }) + + It("flags a variant naming an entry that declares variants itself", func() { + entries := variantFixture( + entryWithVariants("base", "u://base", gallery.Variant{Model: "nested"}), + entryWithVariants("nested", "u://nested", gallery.Variant{Model: "a"}), + plainEntry("a", "u://a"), + ) + + violations := checkVariantReferences(entries) + Expect(violations).To(HaveLen(1)) + Expect(violations[0].Variant).To(Equal("nested")) + Expect(violations[0].Detail).To(ContainSubstring("nesting is not allowed")) + }) + + It("reports every breach in one pass rather than stopping at the first", func() { + entries := variantFixture( + entryWithVariants("base", "u://base", + gallery.Variant{Model: "ghost"}, + gallery.Variant{Model: "phantom"}, + ), + ) + + Expect(checkVariantReferences(entries)).To(HaveLen(2)) + }) + }) + +}) + +var _ = Describe("gallery/index.yaml variant invariants", Ordered, func() { + var entries []gallery.GalleryModel + + BeforeAll(func() { + var err error + entries, err = loadGalleryIndex() + Expect(err).ToNot(HaveOccurred()) + // A truncated or emptied index unmarshals cleanly and would make every + // spec below vacuously pass. + Expect(entries).ToNot(BeEmpty()) + }) + + It("references only existing entries that declare no variants themselves", func() { + v := checkVariantReferences(entries) + Expect(v).To(BeEmpty(), formatViolations(v)) + }) + + // Gallery-wide, not variants-only. The rule this replaced was scoped to + // variant targets because the index carried nine unrelated entries it would + // have failed on. Those nine declared overrides and files but no url, and + // they install cleanly on an empty base now, so there is nothing left to + // exempt and the gate covers the whole catalog in one step. + It("contains no entry that would install nothing", func() { + v := checkEntriesInstallSomething(entries) + Expect(v).To(BeEmpty(), formatViolations(v)) + }) +}) + +// The lint rules above check the catalog as text. This drives the real +// resolution path for the entry a user actually clicked and failed to install, +// so the fix is proven at the layer that broke and not only at the layer that +// should have caught it. +// +// It is a container of its own rather than another spec beside the lint rules +// because an Ordered container stops at its first failure: sharing one would +// let a lint breach skip these silently, which is precisely the kind of +// vacuously-green spec this file exists to avoid. +var _ = Describe("gallery/index.yaml deepseek-v4-flash resolution", Ordered, func() { + var entries []gallery.GalleryModel + var models []*gallery.GalleryModel + var entry *gallery.GalleryModel + + BeforeAll(func() { + var err error + entries, err = loadGalleryIndex() + Expect(err).ToNot(HaveOccurred()) + Expect(entries).ToNot(BeEmpty()) + }) + + BeforeEach(func() { + models = make([]*gallery.GalleryModel, 0, len(entries)) + for i := range entries { + models = append(models, &entries[i]) + } + entry = gallery.FindGalleryElement(models, "deepseek-v4-flash") + Expect(entry).ToNot(BeNil()) + Expect(entry.HasVariants()).To(BeTrue()) + }) + + // The unpinned pass covers the entry the user clicks. It does NOT prove the + // variants are installable: with no probe wired every size is unknown and + // the ranking ties, so the base wins and this only ever exercises the + // parent's own url. The pin spec below is what covers the four targets. + // + // Note that a base selection reports Variant.Model as the ENTRY's name + // rather than as empty, so asserting a non-empty Model here would look like + // a check that a declared variant won while passing on the base every time. + It("yields an installable entry for the entry itself", func() { + env := gallery.ResolveEnv{ + AvailableMemory: 512 << 30, + BackendCompatible: func(string) bool { return true }, + } + + resolved, selected, err := gallery.ResolveVariant(models, entry, env, "") + Expect(err).ToNot(HaveOccurred()) + Expect(entryInstallsSomething(*resolved)).To(BeTrue(), + "resolved entry %q (variant %q) carries no payload, so InstallModelFromGallery would refuse it", + resolved.Name, selected.Model) + }) + + // Pinning reaches each target directly, which is what actually proves all + // four are installable: every one of them is resolved, not just whichever + // the ranking happens to prefer. + // + // None of the four declares a url: any more. They describe themselves + // entirely through overrides: and files:, which applyModel lays over an + // empty base, so this also pins that dropping the urls left them + // installable. + It("yields an installable entry for every declared variant pin", func() { + env := gallery.ResolveEnv{ + AvailableMemory: 512 << 30, + BackendCompatible: func(string) bool { return true }, + } + + for _, v := range entry.Variants { + resolved, _, err := gallery.ResolveVariant(models, entry, env, v.Model) + Expect(err).ToNot(HaveOccurred(), "pinning %q", v.Model) + Expect(resolved.URL).To(BeEmpty(), "variant %q should reach the empty-base path, not a fetch", v.Model) + Expect(entryInstallsSomething(*resolved)).To(BeTrue(), "variant %q carries no payload", v.Model) + } + }) +}) diff --git a/core/http/endpoints/localai/gallery.go b/core/http/endpoints/localai/gallery.go index fabae336d786..fb0a77bb4e1d 100644 --- a/core/http/endpoints/localai/gallery.go +++ b/core/http/endpoints/localai/gallery.go @@ -25,6 +25,11 @@ type ModelGalleryEndpointService struct { type GalleryModel struct { ID string `json:"id"` + // Variant installs one specific build of an entry that declares variants, + // named as it appears in the entry's `variants` list (see the `variants` + // and `auto_variant` fields of the gallery listing). Leave it empty to let + // LocalAI auto-select the largest build this host can actually run. + Variant string `json:"variant,omitempty"` gallery.GalleryModel } @@ -86,6 +91,7 @@ func (mgs *ModelGalleryEndpointService) ApplyModelGalleryEndpoint() echo.Handler Req: input.GalleryModel, ID: uuid.String(), GalleryElementName: input.ID, + Variant: input.Variant, Galleries: mgs.galleries, BackendGalleries: mgs.backendGalleries, } diff --git a/core/http/react-ui/e2e/backends-management.spec.js b/core/http/react-ui/e2e/backends-management.spec.js index c34d505338fe..11b1791891ac 100644 --- a/core/http/react-ui/e2e/backends-management.spec.js +++ b/core/http/react-ui/e2e/backends-management.spec.js @@ -26,3 +26,54 @@ test.describe('Backends management page', () => { await expect(page.getByPlaceholder('oci://quay.io/example/backend:latest')).toBeVisible() }) }) + +// Backend gallery descriptions are Markdown too: 40 of the entries in +// backend/index.yaml carry headings, inline code, lists or links, and they used +// to be dumped raw into the truncated table cell. +const MARKDOWN_DESCRIPTION = + '# InsightFace\n\nUse `insightface` for face analysis. See [the docs](https://example.com/docs) for **details**.' +const STRIPPED_DESCRIPTION = + 'InsightFace Use insightface for face analysis. See the docs for details.' + +test.describe('Backends management page - Markdown descriptions', () => { + test.beforeEach(async ({ page }) => { + await page.route('**/api/backends*', (route) => { + route.fulfill({ + contentType: 'application/json', + body: JSON.stringify({ + backends: [ + { name: 'markdown-backend', description: MARKDOWN_DESCRIPTION, installed: false }, + { name: 'plain-backend', description: '', installed: false }, + ], + }), + }) + }) + await page.goto('/app/backends') + await expect(page.locator('th', { hasText: 'Description' })).toBeVisible({ timeout: 10_000 }) + }) + + test('table cell shows the description as clean text, not raw Markdown', async ({ page }) => { + const cell = page.locator('tr', { hasText: 'markdown-backend' }).locator('span[title]', { hasText: 'InsightFace' }) + + await expect(cell).toHaveText(STRIPPED_DESCRIPTION) + // The syntax itself must be gone, not merely rendered somewhere. + await expect(cell).not.toContainText('#') + await expect(cell).not.toContainText('`') + await expect(cell).not.toContainText('**') + await expect(cell).not.toContainText('https://example.com/docs') + // A block element here would blow up the row height. + await expect(cell.locator('h1')).toHaveCount(0) + }) + + test('title tooltip carries the stripped text, not raw Markdown', async ({ page }) => { + const cell = page.locator('tr', { hasText: 'markdown-backend' }).locator('span[title]', { hasText: 'InsightFace' }) + + await expect(cell).toHaveAttribute('title', STRIPPED_DESCRIPTION) + }) + + test('a backend with no description still shows the placeholder', async ({ page }) => { + const row = page.locator('tr', { hasText: 'plain-backend' }) + + await expect(row.locator('span[title=""]')).toHaveText('-') + }) +}) diff --git a/core/http/react-ui/e2e/models-gallery.spec.js b/core/http/react-ui/e2e/models-gallery.spec.js index d45b5eedac99..6ec1045beadc 100644 --- a/core/http/react-ui/e2e/models-gallery.spec.js +++ b/core/http/react-ui/e2e/models-gallery.spec.js @@ -8,6 +8,10 @@ const MOCK_MODELS_RESPONSE = { backend: "llama-cpp", installed: false, tags: ["chat"], + // The listing carries only the declaration flag. Describing variants + // costs the server a network probe each, so the description lives + // behind /api/models/variants/:id and is fetched on demand. + has_variants: true, }, { name: "whisper-model", @@ -173,7 +177,9 @@ test.describe("Models Gallery - Backend Features", () => { // The detail view should show Backend label and value const detail = page.locator('td[colspan="8"]'); await expect(detail.locator("text=Backend")).toBeVisible(); - await expect(detail.locator("text=llama-cpp")).toBeVisible(); + // The Backend DetailRow renders before the Variants section, which lists a + // per-variant backend badge of its own, so scope to the first match. + await expect(detail.locator("text=llama-cpp").first()).toBeVisible(); }); }); @@ -433,3 +439,1285 @@ test.describe("Models Gallery - Empty State", () => { await expect(page.locator("tr", { hasText: "llama-model" })).toBeVisible(); }); }); + +// The variant description the companion endpoint returns for llama-model. +// memory_bytes is omitempty server-side, so the mlx variant deliberately +// carries no key at all: the UI must render that as unknown, never 0 B. +// +// quantization and features are omitempty for the same reason. The mlx build +// carries neither, standing in for a backend served from a directory of +// weights whose name declares no format: those cells must degrade to a stated +// "unknown", never to a blank or an "undefined". +const MOCK_VARIANTS_RESPONSE = { + variants: [ + { + model: "llama-model", + backend: "llama-cpp", + memory_bytes: 4 * 1024 * 1024 * 1024, + fits: true, + is_base: true, + quantization: "Q4_K_M", + }, + { + model: "llama-model-q8", + backend: "llama-cpp", + memory_bytes: 8 * 1024 * 1024 * 1024, + fits: true, + is_base: false, + quantization: "Q8_0", + features: ["dflash"], + }, + { + model: "llama-model-mlx", + backend: "mlx", + fits: true, + is_base: false, + }, + { + model: "llama-model-f16", + backend: "llama-cpp", + memory_bytes: 40 * 1024 * 1024 * 1024, + fits: false, + is_base: false, + quantization: "F16", + }, + ], + auto_selected: "llama-model-q8", +}; + +test.describe("Models Gallery - Variant picker", () => { + // installUrls records every install request so a test can assert both the + // presence and the absence of the ?variant= parameter. + let installUrls; + // variantUrls records every companion-endpoint request. It is what proves + // the description is fetched lazily and cached, rather than being paid for + // by every row on page load. + let variantUrls; + // Held requests let a test observe the in-flight state rather than racing it. + let releaseVariants; + + test.beforeEach(async ({ page }) => { + installUrls = []; + variantUrls = []; + releaseVariants = null; + await page.route("**/api/models*", (route) => { + route.fulfill({ + contentType: "application/json", + body: JSON.stringify(MOCK_MODELS_RESPONSE), + }); + }); + await page.route("**/api/models/install/**", (route) => { + installUrls.push(route.request().url()); + return route.fulfill({ + status: 200, + contentType: "application/json", + body: JSON.stringify({ jobID: "variant-install" }), + }); + }); + await page.route("**/api/models/variants/**", async (route) => { + variantUrls.push(route.request().url()); + if (releaseVariants) await releaseVariants; + return route.fulfill({ + status: 200, + contentType: "application/json", + body: JSON.stringify(MOCK_VARIANTS_RESPONSE), + }); + }); + await page.goto("/app/models"); + await expect(page.locator("th", { hasText: "Backend" })).toBeVisible({ + timeout: 10_000, + }); + }); + + const variantRow = (page) => page.locator("tr", { hasText: "llama-model" }).first(); + const plainRow = (page) => + page.locator("tr", { hasText: "stablediffusion-model" }).first(); + const openMenu = (page) => + variantRow(page).getByRole("button", { name: "Choose a variant" }).click(); + + test("the listing alone fetches no variant descriptions", async ({ page }) => { + // The whole point of the companion endpoint: a page load costs zero + // probes no matter how many entries declare variants. + await expect(page.locator("tbody tr").first()).toBeVisible(); + expect(variantUrls).toHaveLength(0); + }); + + test("an entry that declares variants shows the split-button chevron", async ({ + page, + }) => { + await expect( + variantRow(page).getByRole("button", { name: "Choose a variant" }), + ).toBeVisible(); + }); + + test("an entry without variants renders no chevron", async ({ page }) => { + await expect( + plainRow(page).getByRole("button", { name: "Choose a variant" }), + ).toHaveCount(0); + // and still offers an ordinary install + await expect( + plainRow(page).locator("button.btn-primary"), + ).toHaveCount(1); + }); + + test("an entry without variants fetches nothing even when expanded", async ({ + page, + }) => { + await plainRow(page).click(); + await expect(page.locator('td[colspan="8"]')).toBeVisible(); + expect(variantUrls).toHaveLength(0); + }); + + test("plain Install sends no variant parameter", async ({ page }) => { + await plainRow(page).locator("button.btn-primary").click(); + await expect.poll(() => installUrls.length).toBe(1); + expect(installUrls[0]).not.toContain("variant="); + }); + + test("opening the menu fetches the description once and caches it", async ({ + page, + }) => { + await openMenu(page); + await expect(page.locator(".action-menu")).toBeVisible(); + await expect.poll(() => variantUrls.length).toBe(1); + expect(variantUrls[0]).toContain("/api/models/variants/llama-model"); + + // Close and reopen: the cached answer must be reused. + await page.keyboard.press("Escape"); + await openMenu(page); + await expect( + page.locator(".action-menu__item", { hasText: "llama-model-q8" }), + ).toBeVisible(); + expect(variantUrls).toHaveLength(1); + }); + + test("the menu shows a loading state while the description is in flight", async ({ + page, + }) => { + let unblock; + releaseVariants = new Promise((resolve) => { + unblock = resolve; + }); + await openMenu(page); + await expect(page.locator(".action-menu")).toContainText("Loading variants"); + unblock(); + await expect( + page.locator(".action-menu__item", { hasText: "llama-model-q8" }), + ).toBeVisible(); + await expect(page.locator(".action-menu")).not.toContainText( + "Loading variants", + ); + }); + + test("the auto-selected variant is marked in the menu", async ({ page }) => { + await openMenu(page); + const menu = page.locator(".action-menu"); + await expect(menu).toBeVisible(); + const autoItem = menu.locator(".action-menu__item", { + hasText: "llama-model-q8", + }); + await expect(autoItem.locator(".badge", { hasText: "Auto" })).toBeVisible(); + // the base build is identifiable too + await expect( + menu + .locator(".action-menu__item", { hasText: "llama-model" }) + .first() + .locator(".badge", { hasText: "Base build" }), + ).toBeVisible(); + }); + + test("a variant with no memory_bytes renders as unknown, not 0", async ({ + page, + }) => { + await openMenu(page); + const mlxItem = page.locator(".action-menu__item", { + hasText: "llama-model-mlx", + }); + await expect(mlxItem).toContainText("Unknown size"); + await expect(mlxItem).not.toContainText("0 B"); + }); + + test("a variant that does not fit is still selectable", async ({ page }) => { + await openMenu(page); + const f16 = page.locator(".action-menu__item", { + hasText: "llama-model-f16", + }); + await expect(f16.locator(".badge", { hasText: "Does not fit" })).toBeVisible(); + await expect(f16).toBeEnabled(); + }); + + test("choosing a specific variant sends ?variant= on the install", async ({ + page, + }) => { + await openMenu(page); + await page + .locator(".action-menu__item", { hasText: "llama-model-mlx" }) + .click(); + await expect.poll(() => installUrls.length).toBe(1); + expect(installUrls[0]).toContain("variant=llama-model-mlx"); + }); + + test("the expanded detail row lists every variant", async ({ page }) => { + await variantRow(page).click(); + const detail = page.locator('td[colspan="8"]'); + await expect(detail).toContainText("Variants"); + await expect(detail).toContainText("llama-model-q8"); + await expect(detail).toContainText("llama-model-mlx"); + await expect(detail).toContainText("llama-model-f16"); + await expect(detail).toContainText("Unknown size"); + await expect(detail).toContainText("Auto-selected"); + await expect(detail).toContainText("Base build"); + await expect(detail).toContainText("Does not fit"); + await expect(detail).toContainText("mlx"); + // Expanding is the second trigger point, so it pays for exactly one fetch. + expect(variantUrls).toHaveLength(1); + }); + + test("the variant rows line up as columns", async ({ page }) => { + await variantRow(page).click(); + const rows = page.locator(".variant-row"); + await expect(rows).toHaveCount(4); + const columns = await rows.evaluateAll((els) => + els.map((el) => ({ + backend: el.querySelector(".variant-row__backend").getBoundingClientRect().x, + size: el.querySelector(".variant-row__size").getBoundingClientRect().right, + })), + ); + // Names differ in length, so without shared tracks each row would start + // its backend at a different x. Sub-pixel rounding is the only tolerance. + for (const c of columns) { + expect(Math.abs(c.backend - columns[0].backend)).toBeLessThan(1.5); + expect(Math.abs(c.size - columns[0].size)).toBeLessThan(1.5); + } + }); + + test("only the informative status is badged", async ({ page }) => { + await variantRow(page).click(); + const detail = page.locator('td[colspan="8"]'); + await expect(detail.locator(".variant-row")).toHaveCount(4); + // "Fits" was true of three rows out of four and said nothing; the row that + // does not fit is the one worth marking. + await expect(detail.getByText("Fits", { exact: true })).toHaveCount(0); + const unfit = detail.locator(".variant-row--unfit"); + await expect(unfit).toHaveCount(1); + await expect(unfit).toContainText("llama-model-f16"); + await expect(unfit.locator(".badge-warning")).toHaveText("Does not fit"); + // Auto-selected still answers "what do I get if I just hit Install". + await expect( + detail.locator(".variant-row", { hasText: "llama-model-q8" }), + ).toContainText("Auto-selected"); + }); + + test("clicking a variant row installs that variant", async ({ page }) => { + await variantRow(page).click(); + await page + .locator(".variant-row", { hasText: "llama-model-mlx" }) + .click(); + await expect.poll(() => installUrls.length).toBe(1); + expect(installUrls[0]).toContain("variant=llama-model-mlx"); + }); + + test("the menu names each build's quantization alongside backend and size", async ({ + page, + }) => { + // Without it the meta line reads "llama-cpp - 8 GB" for two builds that + // differ entirely in precision, which describes nothing the user is + // choosing between. + await openMenu(page); + await expect( + page.locator(".action-menu__item", { hasText: "llama-model-q8" }), + ).toContainText("llama-cpp · Q8_0 · 8 GB"); + }); + + test("the menu marks a build that serves faster", async ({ page }) => { + // A compact marker, not a sentence: the dropdown has room for the token + // and the detail row carries the spelled-out name. + await openMenu(page); + await expect( + page + .locator(".action-menu__item", { hasText: "llama-model-q8" }) + .locator(".badge", { hasText: "DFLASH" }), + ).toBeVisible(); + }); + + test("a build naming no quantization drops the segment rather than blanking", async ({ + page, + }) => { + // The degrade contract in the compact surface: no empty segment, no + // dangling separator, and above all no "undefined". + await openMenu(page); + const item = page.locator(".action-menu__item", { + hasText: "llama-model-mlx", + }); + await expect(item).toContainText("mlx · Unknown size"); + await expect(item).not.toContainText("undefined"); + await expect(item).not.toContainText("· ·"); + }); + + test("the detail row gives quantization its own column", async ({ page }) => { + await variantRow(page).click(); + const detail = page.locator(".variant-list"); + + await expect( + detail.locator(".variant-row", { hasText: "llama-model-q8" }).locator(".variant-row__quant"), + ).toHaveText("Q8_0"); + await expect( + detail.locator(".variant-row", { hasText: "llama-model-f16" }).locator(".variant-row__quant"), + ).toHaveText("F16"); + }); + + test("the detail row states an unknown quantization rather than leaving a gap", async ({ + page, + }) => { + // An empty cell in an aligned column reads as a rendering fault, so the + // absent case is spelled out and styled as the exception it is. + await variantRow(page).click(); + const cell = page + .locator(".variant-row", { hasText: "llama-model-mlx" }) + .locator(".variant-row__quant"); + + await expect(cell).toHaveText("Unknown format"); + await expect(cell).toHaveClass(/variant-row__quant--unknown/); + }); + + test("the detail row spells out the serving feature", async ({ page }) => { + // This is the room the detail row has over the dropdown: "DFLASH" names + // nothing to a user who has not met it. + await variantRow(page).click(); + + await expect( + page + .locator(".variant-row", { hasText: "llama-model-q8" }) + .locator(".badge", { hasText: "Faster: DFlash" }), + ).toBeVisible(); + // A build declaring no feature carries no feature badge at all. + await expect( + page + .locator(".variant-row", { hasText: "llama-model-mlx" }) + .locator(".badge", { hasText: "Faster" }), + ).toHaveCount(0); + }); + + test("a variant row is reachable and actionable from the keyboard", async ({ + page, + }) => { + await variantRow(page).click(); + const row = page.locator(".variant-row", { hasText: "llama-model-f16" }); + await row.focus(); + // A build that does not fit stays installable: the explicit choice is an + // override the server honours. + await expect(row).toBeFocused(); + await page.keyboard.press("Enter"); + await expect.poll(() => installUrls.length).toBe(1); + expect(installUrls[0]).toContain("variant=llama-model-f16"); + }); +}); + +// The gallery entries behind two of llama-model's variants, as the listing +// returns them when asked for one by exact name. Every field is deliberately +// unlike the parent's, so a test asserting on them proves the panel resolved +// the variant's own entry rather than re-rendering the row it sits under. +// +// llama-model-mlx is absent on purpose: it stands for a name the listing no +// longer returns, which is a real outcome once a gallery is reloaded between +// describing an entry's variants and asking about one of them. +const VARIANT_ENTRIES = { + "llama-model-q8": { + name: "llama-model-q8", + description: "The eight-bit build, kept for quality-sensitive work.", + backend: "llama-cpp", + installed: false, + license: "q8-only-licence", + tags: ["chat", "q8-only-tag"], + urls: ["https://example.invalid/llama-model-q8"], + additionalFiles: [ + { + filename: "llama-model-q8.gguf", + uri: "https://example.invalid/q8.gguf", + sha256: "q8", + }, + ], + }, + "llama-model-f16": { + name: "llama-model-f16", + description: "The full-precision build.", + backend: "llama-cpp", + installed: false, + license: "f16-only-licence", + tags: ["chat"], + urls: [], + }, +}; + +// The variant list answers "how do these differ". This answers "tell me +// everything about this one", for a build that has no listing row of its own +// while the collapse is on and so is unreachable anywhere else in the page. +test.describe("Models Gallery - Variant details", () => { + let installUrls; + // Requests for a single variant's gallery entry, told apart from the + // gallery's own listing by the page size the detail lookup asks for. + let detailUrls; + + test.beforeEach(async ({ page }) => { + installUrls = []; + detailUrls = []; + + await page.route("**/api/models*", (route) => { + const url = new URL(route.request().url()); + const term = (url.searchParams.get("term") || "").trim(); + const isDetailLookup = + url.pathname.endsWith("/api/models") && + url.searchParams.get("items") === "100" && + term !== ""; + if (isDetailLookup) { + detailUrls.push(url.toString()); + const entry = VARIANT_ENTRIES[term]; + return route.fulfill({ + contentType: "application/json", + body: JSON.stringify({ + ...MOCK_MODELS_RESPONSE, + models: entry ? [entry] : [], + }), + }); + } + return route.fulfill({ + contentType: "application/json", + body: JSON.stringify(MOCK_MODELS_RESPONSE), + }); + }); + await page.route("**/api/models/install/**", (route) => { + installUrls.push(route.request().url()); + return route.fulfill({ + status: 200, + contentType: "application/json", + body: JSON.stringify({ jobID: "variant-install" }), + }); + }); + await page.route("**/api/models/variants/**", (route) => + route.fulfill({ + status: 200, + contentType: "application/json", + body: JSON.stringify(MOCK_VARIANTS_RESPONSE), + }), + ); + await page.goto("/app/models"); + await expect(page.locator("th", { hasText: "Backend" })).toBeVisible({ + timeout: 10_000, + }); + // Expanding the parent is what puts the variant list on screen. + await page.locator("tr", { hasText: "llama-model" }).first().click(); + await expect(page.locator(".variant-row")).toHaveCount(4); + }); + + // Exact, because the names are prefixes of one another: "llama-model" is a + // substring of every other variant's control. + const infoFor = (page, variant) => + page.getByRole("button", { + name: `Show full details for ${variant}`, + exact: true, + }); + + test("every variant carries its own details control", async ({ page }) => { + await expect(page.locator(".variant-row__info")).toHaveCount(4); + // The accessible name has to say which build it acts on: a column of + // identical "info" buttons is useless to a screen reader user. + for (const variant of [ + "llama-model", + "llama-model-q8", + "llama-model-mlx", + "llama-model-f16", + ]) { + await expect(infoFor(page, variant)).toBeVisible(); + } + }); + + test("no details are fetched until the control is used", async ({ page }) => { + // Expanding lists four variants. If the panel prefetched, this would be + // four requests for data nobody has asked to see. + expect(detailUrls).toHaveLength(0); + await infoFor(page, "llama-model-q8").click(); + await expect.poll(() => detailUrls.length).toBe(1); + expect(detailUrls[0]).toContain("term=llama-model-q8"); + }); + + test("the details shown are the variant's own entry, not the parent's", async ({ + page, + }) => { + await infoFor(page, "llama-model-q8").click(); + const panel = page.locator(".variant-detail"); + await expect(panel).toContainText( + "The eight-bit build, kept for quality-sensitive work.", + ); + await expect(panel).toContainText("q8-only-licence"); + await expect(panel).toContainText("q8-only-tag"); + await expect(panel).toContainText("https://example.invalid/llama-model-q8"); + await expect(panel).toContainText("1 file"); + // The parent's own description renders in the same expanded row. If the + // panel were re-rendering the parent, this would be here too. + await expect(panel).not.toContainText("A llama model"); + }); + + test("a variant's details never nest another variants list", async ({ + page, + }) => { + // Two levels of disclosure is already deep. A picker inside a picker is + // where it stops being legible. + await infoFor(page, "llama-model-q8").click(); + await expect(page.locator(".variant-detail")).toBeVisible(); + await expect(page.locator(".variant-detail .variant-list")).toHaveCount(0); + }); + + test("opening the details does not install anything", async ({ page }) => { + await infoFor(page, "llama-model-q8").click(); + await expect(page.locator(".variant-detail")).toContainText( + "q8-only-licence", + ); + // The panel is fully rendered by now, so an install triggered by the same + // click would have fired. + expect(installUrls).toHaveLength(0); + }); + + test("the variant row still installs with the control alongside it", async ({ + page, + }) => { + await page.locator(".variant-row", { hasText: "llama-model-q8" }).click(); + await expect.poll(() => installUrls.length).toBe(1); + expect(installUrls[0]).toContain("variant=llama-model-q8"); + // And installing is not a request for details either. + expect(detailUrls).toHaveLength(0); + }); + + test("a variant whose entry cannot be resolved says so", async ({ page }) => { + await infoFor(page, "llama-model-mlx").click(); + const panel = page.locator(".variant-detail"); + // Visibly degraded, not silently blank: an empty panel reads as a + // rendering fault rather than as a lookup that came back with nothing. + await expect(panel).toContainText( + "Details for llama-model-mlx could not be loaded.", + ); + await expect(panel.locator(".variant-detail__state--error")).toBeVisible(); + }); + + test("the details are fetched once and reused", async ({ page }) => { + await infoFor(page, "llama-model-q8").click(); + await expect.poll(() => detailUrls.length).toBe(1); + await page + .getByRole("button", { + name: "Hide full details for llama-model-q8", + exact: true, + }) + .click(); + await expect(page.locator(".variant-detail")).toHaveCount(0); + await infoFor(page, "llama-model-q8").click(); + await expect(page.locator(".variant-detail")).toContainText( + "q8-only-licence", + ); + expect(detailUrls).toHaveLength(1); + }); + + test("only one variant's details are open at a time", async ({ page }) => { + // The list is a comparison; two open panels push the rows being compared + // apart. + await infoFor(page, "llama-model-q8").click(); + await infoFor(page, "llama-model-f16").click(); + await expect(page.locator(".variant-detail")).toHaveCount(1); + await expect(page.locator(".variant-detail")).toContainText( + "f16-only-licence", + ); + }); + + test("the control is keyboard reachable, activates, and dismisses", async ({ + page, + }) => { + const info = infoFor(page, "llama-model-q8"); + await info.focus(); + await expect(info).toBeFocused(); + // A visible focus indicator, not merely a focused element. + await expect(info).toHaveCSS("outline-style", "solid"); + await page.keyboard.press("Enter"); + await expect(page.locator(".variant-detail")).toContainText( + "q8-only-licence", + ); + await expect( + page.getByRole("button", { + name: "Hide full details for llama-model-q8", + exact: true, + }), + ).toHaveAttribute("aria-expanded", "true"); + + await page.keyboard.press("Escape"); + await expect(page.locator(".variant-detail")).toHaveCount(0); + // Focus comes back to the control that opened it, rather than being + // dropped at the top of the document. + await expect(infoFor(page, "llama-model-q8")).toBeFocused(); + }); + + test("the variant rows still line up with the control in front of them", async ({ + page, + }) => { + // The extra column must be shared like every other, or the names it sits + // beside stop forming a column. + const rows = page.locator(".variant-row"); + const columns = await rows.evaluateAll((els) => + els.map((el) => ({ + name: el.querySelector(".variant-row__name").getBoundingClientRect().x, + size: el + .querySelector(".variant-row__size") + .getBoundingClientRect().right, + })), + ); + for (const c of columns) { + expect(Math.abs(c.name - columns[0].name)).toBeLessThan(1.5); + expect(Math.abs(c.size - columns[0].size)).toBeLessThan(1.5); + } + }); +}); + +// The collapsed view is the deduplicated gallery: every entry installable in +// its own right, with nothing shown twice. Here whisper-model stands in for a +// build llama-model already offers as a variant, so it is the only row that +// drops; stablediffusion-model is nobody's variant and stays. The filter is +// server-side because the listing paginates, so these specs assert on the +// request the page actually sends, not just on the rows it renders. +const COLLAPSED_RESPONSE = { + ...MOCK_MODELS_RESPONSE, + models: MOCK_MODELS_RESPONSE.models.filter((m) => m.name !== "whisper-model"), + availableModels: 2, + totalPages: 1, + currentPage: 1, +}; + +// What a search for the grouped-away build gets back with the collapse off: +// the build itself. +const SEARCH_HIT_RESPONSE = { + ...MOCK_MODELS_RESPONSE, + models: MOCK_MODELS_RESPONSE.models.filter((m) => m.name === "whisper-model"), + availableModels: 1, + totalPages: 1, + currentPage: 1, +}; + +// And with the collapse on: the same term matched against the same builds, but +// the match reported at the entry that offers it, which is the row the user can +// act on. One row, and a count that says one. +const SEARCH_PARENT_RESPONSE = { + ...MOCK_MODELS_RESPONSE, + models: MOCK_MODELS_RESPONSE.models.filter((m) => m.name === "llama-model"), + availableModels: 1, + totalPages: 1, + currentPage: 1, +}; + +test.describe("Models Gallery - Collapsed Listing", () => { + let listingUrls; + + test.beforeEach(async ({ page }) => { + listingUrls = []; + + await page.route("**/api/models*", (route) => { + const url = new URL(route.request().url()); + // Only the gallery's own listing. Sibling routes like + // /api/models/estimate share the prefix, and the recommended-models + // panel queries /api/models itself with its own page size, so neither + // must pollute the record of what the page sent, nor pick up the + // narrowed bodies below. + const isListing = + url.pathname.endsWith("/api/models") && + url.searchParams.get("items") === "9"; + if (isListing) { + listingUrls.push(url); + } + const term = (url.searchParams.get("term") || "").trim(); + const collapsed = url.searchParams.get("collapse_variants") === "true"; + const tag = url.searchParams.get("tag"); + // Stands in for the server: the term is matched against every build + // either way, and the collapse decides how a match is reported. Grouped, + // a hit on a build a parent already offers comes back as that parent; + // ungrouped, it comes back as itself. + let body = collapsed ? COLLAPSED_RESPONSE : MOCK_MODELS_RESPONSE; + if (isListing && term === "whisper-model") + body = collapsed ? SEARCH_PARENT_RESPONSE : SEARCH_HIT_RESPONSE; + // A term matching no entry, so the empty state is reachable from a + // search as well as from a chip and the two can be told apart. + else if (isListing && term) body = EMPTY_FILTERED_RESPONSE; + // A usecase filter matches nothing in this fixture, so the empty state + // stays reachable and the specs can pin down what it says. + if (isListing && tag) body = EMPTY_FILTERED_RESPONSE; + route.fulfill({ + contentType: "application/json", + body: JSON.stringify(body), + }); + }); + + await page.goto("/app/models"); + await expect(page.locator("th", { hasText: "Backend" })).toBeVisible({ + timeout: 10_000, + }); + }); + + // The house pattern for these toggles: the checkbox itself is a zero-sized + // opacity-0 input, so state is read through the wrapping label and changed + // by clicking the visible track. + const collapseToggle = (page) => page.getByLabel("One row per model"); + const flipCollapse = (page) => + page.getByTestId("models-collapse-variants").locator(".toggle__track").click(); + + test("the collapse toggle sits in the refinements band, on by default", async ({ + page, + }) => { + // It belongs with the other narrowing controls rather than among the + // taxonomy chips: it refines a listing the user is already reading. + await expect( + page + .getByTestId("models-filters-refine") + .getByTestId("models-collapse-variants"), + ).toBeVisible(); + await expect( + page.getByTestId("models-collapse-variants"), + ).toContainText("One row per model"); + // Default collapsed, so the default view is one row per model. + await expect(collapseToggle(page)).toBeChecked(); + }); + + test("turning the toggle off reveals the builds the collapse hid", async ({ + page, + }) => { + // Browsing, as opposed to finding. Search reaches a build whose name you + // already know; only this enumerates every build the gallery holds. + await expect(page.locator("tr", { hasText: "whisper-model" })).toHaveCount(0); + + await flipCollapse(page); + + await expect(page.locator("tr", { hasText: "whisper-model" })).toBeVisible(); + // Off means the parameter is absent, so opting out asks for exactly the + // listing every other API client gets. + await expect + .poll(() => + listingUrls[listingUrls.length - 1].searchParams.get( + "collapse_variants", + ), + ) + .toBeNull(); + }); + + test("changing the toggle resets to page 1", async ({ page }) => { + await flipCollapse(page); + + await expect + .poll(() => listingUrls[listingUrls.length - 1].searchParams.get("page")) + .toBe("1"); + }); + + test("the choice survives a reload", async ({ page }) => { + await flipCollapse(page); + await expect(page.locator("tr", { hasText: "whisper-model" })).toBeVisible(); + + await page.reload(); + await expect(page.locator("th", { hasText: "Backend" })).toBeVisible({ + timeout: 10_000, + }); + + await expect(collapseToggle(page)).not.toBeChecked(); + await expect(page.locator("tr", { hasText: "whisper-model" })).toBeVisible(); + }); + + test("browsing collapses: the parent stays, the build it offers drops", async ({ + page, + }) => { + // A filter that kept only the entries declaring variants would wrongly + // drop stablediffusion-model too. + await expect(page.locator("tr", { hasText: "llama-model" })).toBeVisible(); + await expect(page.locator("tr", { hasText: "whisper-model" })).toHaveCount( + 0, + ); + await expect( + page.locator("tr", { hasText: "stablediffusion-model" }), + ).toBeVisible(); + + // Asserted over every listing request, so a first paint that fetched the + // uncollapsed listing before settling would still fail. + expect(listingUrls.length).toBeGreaterThan(0); + for (const url of listingUrls) { + expect(url.searchParams.get("collapse_variants")).toBe("true"); + } + }); + + test("searching a build the collapse groups away surfaces the entry offering it", async ({ + page, + }) => { + // Grouped, a search is still answered, and answered with a row that can be + // acted on. Typing the name of an entry the gallery does hold must never + // produce "no models found", which reads as "that model does not exist"; + // returning the build itself would instead put a row in the listing that + // the view the user asked for has no place for. + await page.locator(".search-bar input").fill("whisper-model"); + + await expect(page.locator("tr", { hasText: "llama-model" })).toBeVisible(); + await expect(page.locator("tr", { hasText: "whisper-model" })).toHaveCount( + 0, + ); + await expect(page.locator(".empty-state")).toHaveCount(0); + }); + + test("the same search returns the build itself with the toggle off", async ({ + page, + }) => { + // The other half of what the toggle now controls. Off, search answers with + // the individual build, exactly as it does for every client that never + // sends the parameter. + await flipCollapse(page); + await page.locator(".search-bar input").fill("whisper-model"); + + await expect(page.locator("tr", { hasText: "whisper-model" })).toBeVisible(); + }); + + test("the search term is sent alongside the collapse, not instead of it", async ({ + page, + }) => { + // The server decides what an active search means. The page keeps asking + // for the collapsed listing so that decision lives in one place, and so + // clearing the box goes straight back to the browsing view. + await page.locator(".search-bar input").fill("whisper-model"); + await expect.poll( + () => listingUrls[listingUrls.length - 1].searchParams.get("term"), + ).toBe("whisper-model"); + + const searched = listingUrls[listingUrls.length - 1]; + expect(searched.searchParams.get("collapse_variants")).toBe("true"); + }); + + test("clearing the search box returns to the collapsed listing", async ({ + page, + }) => { + await page.locator(".search-bar input").fill("whisper-model"); + // The search narrowed to the one surfaced row, so the other browsing rows + // are gone and their return is what proves the term was dropped. + await expect( + page.locator("tr", { hasText: "stablediffusion-model" }), + ).toHaveCount(0); + + await page.locator(".search-bar input").fill(""); + + await expect( + page.locator("tr", { hasText: "stablediffusion-model" }), + ).toBeVisible(); + await expect(page.locator("tr", { hasText: "whisper-model" })).toHaveCount( + 0, + ); + await expect(page.locator("tr", { hasText: "llama-model" })).toBeVisible(); + }); + + test("a legacy '0' in storage is not read as a choice", async ({ page }) => { + // An older build wrote '1'/'0' from an effect that ran on mount, so those + // values record that the page was opened rather than that anyone picked a + // view. Only 'on'/'off' counts, so a legacy visitor gets the default. + await page.evaluate(() => { + localStorage.setItem("localai-models-collapse-variants-filter", "0"); + }); + await page.reload(); + await expect(page.locator("th", { hasText: "Backend" })).toBeVisible({ + timeout: 10_000, + }); + + await expect(page.locator("tr", { hasText: "whisper-model" })).toHaveCount( + 0, + ); + const last = listingUrls[listingUrls.length - 1]; + expect(last.searchParams.get("collapse_variants")).toBe("true"); + }); + + test("searching never dead-ends on the default view", async ({ page }) => { + // The regression 462583f38 existed to prevent, re-checked now that search + // respects the toggle instead of switching it off. A user who never touches + // the control must still get an answer for a build the gallery holds; that + // the answer is the entry offering it is the collapse doing its job, not + // the dead end coming back. + await expect(collapseToggle(page)).toBeChecked(); + + await page.locator(".search-bar input").fill("whisper-model"); + + await expect(page.locator("tr", { hasText: "llama-model" })).toBeVisible(); + await expect(page.locator(".empty-state")).toHaveCount(0); + // Still asked for collapsed: the server decides what a term means. + await expect + .poll(() => + listingUrls[listingUrls.length - 1].searchParams.get( + "collapse_variants", + ), + ) + .toBe("true"); + }); + + test("the empty state does not blame the collapse for a chip", async ({ + page, + }) => { + await page.locator(".filter-btn", { hasText: "Chat" }).click(); + + await expect(page.locator(".empty-state-title")).toHaveText( + "No models found", + ); + await expect(page.locator(".empty-state-text")).toHaveText( + "No models match your current search or filters.", + ); + // The chip is applied server-side over every build the gallery holds, and + // a match there is always reported as some row, so an empty result means + // nothing matched rather than that the collapse swallowed the matches. + // Pointing at the toggle would send the user to a control that cannot + // change this result. + await expect(page.locator(".empty-state-hint")).toHaveCount(0); + }); + + test("the empty state does not blame the collapse for a search", async ({ + page, + }) => { + // Same reasoning as the chip: the term is matched against every build, so + // with one typed the collapse cannot be what emptied the listing. + await page.locator(".search-bar input").fill("nothing-matches-this"); + await expect(page.locator(".empty-state")).toBeVisible(); + + await expect(page.locator(".empty-state-hint")).toHaveCount(0); + }); + + test("clear filters returns to the collapsed browsing view", async ({ + page, + }) => { + await page.locator(".filter-btn", { hasText: "Chat" }).click(); + await expect(page.locator(".empty-state")).toBeVisible(); + + await page.getByRole("button", { name: "Clear filters" }).click(); + + await expect(page.locator("tr", { hasText: "llama-model" })).toBeVisible(); + await expect(page.locator("tr", { hasText: "whisper-model" })).toHaveCount( + 0, + ); + }); + + test("clear filters resets the collapse toggle to its default", async ({ + page, + }) => { + // It is a filter like the others, so leaving it behind would make "clear + // filters" a half-truth. + await flipCollapse(page); + await expect(page.locator("tr", { hasText: "whisper-model" })).toBeVisible(); + await page.locator(".filter-btn", { hasText: "Chat" }).click(); + await expect(page.locator(".empty-state")).toBeVisible(); + + await page.getByRole("button", { name: "Clear filters" }).click(); + + await expect(collapseToggle(page)).toBeChecked(); + await expect(page.locator("tr", { hasText: "whisper-model" })).toHaveCount( + 0, + ); + }); + + test("the clear button appears for the toggle alone", async ({ page }) => { + // Turning the collapse off is a filter change with nothing else set, so + // the empty state must still offer a way back. + await flipCollapse(page); + await page.locator(".filter-btn", { hasText: "Chat" }).click(); + + await expect( + page.getByRole("button", { name: "Clear filters" }), + ).toBeVisible(); + }); +}); + +// Gallery descriptions are third-party Markdown. They used to be dumped raw +// into the UI, so a model whose description opened with an ATX heading showed +// a literal "# Name [](url)" in the list. +const MARKDOWN_DESCRIPTION = + "# Qwen3.6-27B\n\nChat with it at [the Qwen site](https://chat.qwen.ai) for **free**."; +const MARKDOWN_MODELS_RESPONSE = { + ...MOCK_MODELS_RESPONSE, + models: [ + { + name: "markdown-model", + description: MARKDOWN_DESCRIPTION, + backend: "llama-cpp", + installed: false, + tags: ["chat"], + }, + { + name: "headings-model", + description: + "# Top Heading\n\nBody copy.\n\n## Sub Heading\n\nMore body copy.", + backend: "llama-cpp", + installed: false, + tags: ["chat"], + }, + { + name: "no-description-model", + description: "", + backend: "llama-cpp", + installed: false, + tags: ["chat"], + }, + ], + availableModels: 3, + installedModels: 0, +}; + +test.describe("Models Gallery - Markdown descriptions", () => { + test.beforeEach(async ({ page }) => { + await page.route("**/api/models*", (route) => { + route.fulfill({ + contentType: "application/json", + body: JSON.stringify(MARKDOWN_MODELS_RESPONSE), + }); + }); + await page.goto("/app/models"); + await expect(page.locator("th", { hasText: "Backend" })).toBeVisible({ + timeout: 10_000, + }); + }); + + test("table cell shows the description as clean text, not raw Markdown", async ({ + page, + }) => { + const row = page.locator("tr", { hasText: "markdown-model" }); + const cell = row.locator("div[title]", { hasText: "Qwen3.6-27B" }); + + await expect(cell).toHaveText( + "Qwen3.6-27B Chat with it at the Qwen site for free.", + ); + // The syntax itself must be gone, not merely rendered somewhere. + await expect(cell).not.toContainText("#"); + await expect(cell).not.toContainText("[]("); + await expect(cell).not.toContainText("**"); + await expect(cell).not.toContainText("https://chat.qwen.ai"); + // A block element here would blow up the row height. + await expect(cell.locator("h1")).toHaveCount(0); + }); + + test("title tooltip carries the stripped text, not raw Markdown", async ({ + page, + }) => { + const row = page.locator("tr", { hasText: "markdown-model" }); + const cell = row.locator("div[title]", { hasText: "Qwen3.6-27B" }); + + await expect(cell).toHaveAttribute( + "title", + "Qwen3.6-27B Chat with it at the Qwen site for free.", + ); + }); + + test("expanded detail row renders the description as real markup", async ({ + page, + }) => { + await page.locator("tr", { hasText: "markdown-model" }).click(); + + const detail = page.locator('td[colspan="8"]'); + await expect(detail.locator("h1", { hasText: "Qwen3.6-27B" })).toBeVisible(); + const link = detail.locator('a[href="https://chat.qwen.ai"]'); + await expect(link).toBeVisible(); + await expect(link).toHaveText("the Qwen site"); + await expect(detail.locator("strong", { hasText: "free" })).toBeVisible(); + }); + + test("a model without a description still shows the placeholder", async ({ + page, + }) => { + const row = page.locator("tr", { hasText: "no-description-model" }); + await expect(row).toBeVisible(); + await expect(row.locator("div[title='']")).toHaveText("—"); + }); + + test("a heading in the description renders on the UI type scale", async ({ + page, + }) => { + await page.locator("tr", { hasText: "headings-model" }).click(); + const prose = page.locator(".detail-prose__body.markdown-body"); + await expect(prose).toBeVisible(); + + const h1 = prose.locator("h1"); + await expect(h1).toHaveText("Top Heading"); + const sizes = await prose.evaluate((el) => { + const px = (sel) => + parseFloat(getComputedStyle(el.querySelector(sel)).fontSize); + return { h1: px("h1"), h2: px("h2"), p: px("p") }; + }); + // The bug: an unscoped h1 inherits the browser default 2em, which is 26px + // inside this 13px surface and swamps the pane. The scale tops out at + // --text-xl (1.25rem / 20px), so anything at or above that is the default + // leaking through rather than a styled heading. + expect(sizes.h1).toBeGreaterThan(sizes.p); + expect(sizes.h1).toBeLessThanOrEqual(20); + expect(sizes.h1).toBeGreaterThanOrEqual(14); + // The inverse defect: a subheading that is indistinguishable from body + // text. It must stay below h1 and at or above the body size. + expect(sizes.h2).toBeLessThanOrEqual(sizes.h1); + expect(sizes.h2).toBeGreaterThanOrEqual(sizes.p); + }); + + test("the description sits outside the label/value grid on a readable measure", async ({ + page, + }) => { + await page.locator("tr", { hasText: "headings-model" }).click(); + const detail = page.locator('td[colspan="8"]'); + // Description is no longer a row of the scalar table. + await expect(detail.locator("table td", { hasText: "Description" })).toHaveCount( + 0, + ); + await expect(detail.locator(".detail-prose__label")).toHaveText( + "Description", + ); + const proseWidth = await page + .locator(".detail-prose__body") + .evaluate((el) => el.getBoundingClientRect().width); + const paneWidth = await detail.evaluate( + (el) => el.getBoundingClientRect().width, + ); + // A measure, not the full pane: the cap is a ch count, so the exact pixel + // value moves with the font, but it must stay well inside the pane. + expect(proseWidth).toBeLessThan(paneWidth * 0.85); + }); + + test("a model without a description renders no prose block", async ({ + page, + }) => { + await page.locator("tr", { hasText: "no-description-model" }).click(); + const detail = page.locator('td[colspan="8"]'); + await expect(detail).toBeVisible(); + await expect(detail.locator(".detail-prose")).toHaveCount(0); + // The scalar rows still render, so the pane is not blank. + await expect(detail).toContainText("Backend"); + }); +}); + +// The filter block is three deliberate bands: query scope (search + backend +// select), the use-case chip row, and the refinements (fits-in-GPU + context). +// These assert the separation holds, because the regression they guard against +// is the refinements being swept back into the chip row's wrap, where their +// position depends on how many chips happen to wrap at the current width. +test.describe("Models Gallery - Filter layout structure", () => { + test.beforeEach(async ({ page }) => { + await page.route("**/api/models*", (route) => { + route.fulfill({ + contentType: "application/json", + body: JSON.stringify(MOCK_MODELS_RESPONSE), + }); + }); + await page.route("**/api/backends/usecases", (route) => { + route.fulfill({ + contentType: "application/json", + body: JSON.stringify(BACKEND_USECASES_MOCK), + }); + }); + await page.route("**/api/resources", (route) => { + route.fulfill({ + contentType: "application/json", + body: JSON.stringify(MOCK_GPU_RESOURCES_RESPONSE), + }); + }); + await page.goto("/app/models"); + await expect(page.locator("th", { hasText: "Backend" })).toBeVisible({ + timeout: 10_000, + }); + }); + + test("the chip row contains only use-case chips", async ({ page }) => { + const chipRow = page.locator(".filter-bar"); + await expect(chipRow).toHaveCount(1); + // Nothing but .filter-btn children: no toggle, no select, no slider. + const childClasses = await chipRow.evaluate((el) => + Array.from(el.children).map((c) => c.className), + ); + expect(childClasses.length).toBeGreaterThan(0); + for (const cls of childClasses) { + expect(cls).toContain("filter-btn"); + } + await expect(chipRow.locator("input[type='range']")).toHaveCount(0); + await expect(chipRow.locator(".filter-bar-group__toggle")).toHaveCount(0); + await expect(chipRow.getByText("All Backends")).toHaveCount(0); + }); + + test("refinements live in their own band, outside the chip row", async ({ + page, + }) => { + const refine = page.getByTestId("models-filters-refine"); + await expect(refine).toBeVisible(); + await expect(refine.locator(".filter-bar")).toHaveCount(0); + await expect(refine.getByText("Fits in GPU")).toBeVisible(); + await expect(refine.locator("#models-context-size")).toBeVisible(); + // The band is a sibling of the chip row, never a descendant. + const nested = await page + .locator(".filter-bar") + .locator('[data-testid="models-filters-refine"]') + .count(); + expect(nested).toBe(0); + }); + + test("the backend select sits in the query band above the chips", async ({ + page, + }) => { + const selectBtn = page.locator("button", { hasText: "All Backends" }); + await expect(selectBtn).toBeVisible(); + const inChipRow = await page + .locator(".filter-bar") + .locator("button", { hasText: "All Backends" }) + .count(); + expect(inChipRow).toBe(0); + // Reads above the chips it gates. + const selectBox = await selectBtn.boundingBox(); + const chipBox = await page.locator(".filter-bar").boundingBox(); + expect(selectBox.y).toBeLessThan(chipBox.y); + }); + + test("refinements stay grouped and on one band at a narrow width", async ({ + page, + }) => { + await page.setViewportSize({ width: 900, height: 900 }); + const refine = page.getByTestId("models-filters-refine"); + await expect(refine).toBeVisible(); + const chipBox = await page.locator(".filter-bar").boundingBox(); + const refineBox = await refine.boundingBox(); + // Below the chip row, not interleaved with it. + expect(refineBox.y).toBeGreaterThanOrEqual(chipBox.y + chipBox.height - 1); + await expect(refine.getByText("Fits in GPU")).toBeVisible(); + await expect(refine.locator("#models-context-size")).toBeVisible(); + }); + + test("chips expose pressed state and the context slider is labelled", async ({ + page, + }) => { + const chatBtn = page.locator(".filter-btn", { hasText: "Chat" }); + await expect(chatBtn).toHaveAttribute("aria-pressed", "false"); + await chatBtn.click(); + await expect(chatBtn).toHaveAttribute("aria-pressed", "true"); + + const slider = page.locator("#models-context-size"); + // The slider steps over an index, so the announced value must be the size. + await expect(slider).toHaveAttribute("aria-valuetext", /^\d+K$/); + await expect(page.locator("label[for='models-context-size']")).toBeVisible(); + }); + + test("a keyboard-focused chip shows a focus ring", async ({ page }) => { + // The global :focus-visible rule is wrapped in :where(), so it ties with + // .filter-btn on specificity and loses on order. Without an explicit rule + // the chips render their resting shadow while focused, i.e. no indicator. + await page.locator(".filter-bar-group__search input").click(); + await page.keyboard.press("Tab"); // backend select + await page.keyboard.press("Tab"); // first chip + const focused = page.locator(".filter-btn:focus-visible"); + await expect(focused).toHaveCount(1); + // The ring transitions in, so settle before reading the computed value. + await page.waitForTimeout(400); + const shadow = await focused.evaluate( + (el) => getComputedStyle(el).boxShadow, + ); + // A 3px spread ring, not the 1px/2px resting drop shadow. + expect(shadow).toMatch(/0px 0px 0px 3px/); + }); + + test("the context control is keyboard reachable and drives the value", async ({ + page, + }) => { + const slider = page.locator("#models-context-size"); + const before = await slider.inputValue(); + await slider.focus(); + await expect(slider).toBeFocused(); + await page.keyboard.press("ArrowRight"); + await expect(slider).not.toHaveValue(before); + await expect(slider).toHaveAttribute("aria-valuetext", /^\d+K$/); + }); +}); diff --git a/core/http/react-ui/e2e/models-recommended-panel.spec.js b/core/http/react-ui/e2e/models-recommended-panel.spec.js new file mode 100644 index 000000000000..7dad5ed9e1b8 --- /dev/null +++ b/core/http/react-ui/e2e/models-recommended-panel.spec.js @@ -0,0 +1,148 @@ +import { test, expect } from "./coverage-fixtures.js"; + +// The "Recommended for your hardware" strip defaults its own prominence off the +// installed-model count and remembers both the collapse choice and a dismissal, +// so every assertion here is about state that must survive a reload. + +const DISMISS_KEY = "localai-models-recommended-dismissed"; +const COLLAPSE_KEY = "localai-models-recommended-collapsed"; + +const REC_MODELS = [ + { name: "tiny-chat", description: "Tiny", backend: "llama-cpp", installed: false, tags: ["chat"] }, + { name: "small-chat", description: "Small", backend: "llama-cpp", installed: false, tags: ["chat"] }, +]; + +function listResponse(installedModels) { + return { + models: REC_MODELS, + allBackends: ["llama-cpp"], + allTags: ["chat"], + availableModels: REC_MODELS.length, + installedModels, + totalPages: 1, + currentPage: 1, + }; +} + +const ESTIMATES = { + "tiny-chat": { sizeBytes: 512 * 1024 * 1024, sizeDisplay: "512.0 MB", estimates: { 4096: { vramBytes: 700 * 1024 * 1024, vramDisplay: "700.0 MB" } } }, + "small-chat": { sizeBytes: 1024 * 1024 * 1024, sizeDisplay: "1.00 GB", estimates: { 4096: { vramBytes: 1400 * 1024 * 1024, vramDisplay: "1.40 GB" } } }, +}; + +// installedModels drives the panel's default state, so each test picks its own. +async function mockGallery(page, installedModels) { + // Registered first so the more specific routes below take precedence: + // Playwright matches the most recently added handler. + await page.route("**/api/models*", (route) => + route.fulfill({ contentType: "application/json", body: JSON.stringify(listResponse(installedModels)) }), + ); + await page.route("**/api/models/estimate/*", (route) => { + const name = decodeURIComponent(new URL(route.request().url()).pathname.split("/").pop()); + return route.fulfill({ contentType: "application/json", body: JSON.stringify(ESTIMATES[name] || {}) }); + }); + // CPU-only host, which is the branch that shows the "no GPU detected" note. + await page.route("**/api/resources", (route) => + route.fulfill({ contentType: "application/json", body: JSON.stringify({ type: "cpu", available: false, gpus: [] }) }), + ); +} + +const panel = (page) => page.getByTestId("recommended-models"); +const toggle = (page) => page.getByTestId("recommended-models-toggle"); +const grid = (page) => page.locator("#rec-models-content"); + +async function gotoModels(page) { + await page.goto("/app/models"); + await expect(panel(page)).toBeVisible({ timeout: 20_000 }); +} + +test.describe("Models gallery - recommended panel prominence", () => { + test("first visit with nothing installed shows the panel expanded", async ({ page }) => { + await mockGallery(page, 0); + await gotoModels(page); + + await expect(toggle(page)).toHaveAttribute("aria-expanded", "true"); + await expect(grid(page)).toBeVisible(); + await expect(grid(page).getByText("tiny-chat")).toBeVisible(); + }); + + test("a user with models installed gets it collapsed by default", async ({ page }) => { + await mockGallery(page, 12); + await gotoModels(page); + + await expect(toggle(page)).toHaveAttribute("aria-expanded", "false"); + await expect(grid(page)).toBeHidden(); + // Collapsed is a summary, not a removal: the heading stays on the page. + await expect(panel(page).getByText("Recommended for your hardware")).toBeVisible(); + await expect(panel(page).getByText("2 models suggested")).toBeVisible(); + }); + + test("the collapsed summary expands again on activation", async ({ page }) => { + await mockGallery(page, 12); + await gotoModels(page); + + await expect(grid(page)).toBeHidden(); + await toggle(page).click(); + + await expect(toggle(page)).toHaveAttribute("aria-expanded", "true"); + await expect(grid(page)).toBeVisible(); + await expect(page.evaluate((k) => localStorage.getItem(k), COLLAPSE_KEY)).resolves.toBe("0"); + }); + + test("the collapse choice persists across a reload", async ({ page }) => { + await mockGallery(page, 0); + await gotoModels(page); + await expect(grid(page)).toBeVisible(); + + await toggle(page).click(); + await expect(grid(page)).toBeHidden(); + + await page.reload(); + await expect(panel(page)).toBeVisible({ timeout: 20_000 }); + await expect(toggle(page)).toHaveAttribute("aria-expanded", "false"); + await expect(grid(page)).toBeHidden(); + }); + + test("dismissing it persists across a reload", async ({ page }) => { + await mockGallery(page, 0); + await gotoModels(page); + + await panel(page).getByRole("button", { name: "Dismiss recommendations" }).click(); + await expect(panel(page)).toHaveCount(0); + await expect(page.evaluate((k) => localStorage.getItem(k), DISMISS_KEY)).resolves.toBe("1"); + + await page.reload(); + // The table is the marker that the page finished rendering without the panel. + await expect(page.locator("table tbody tr").first()).toBeVisible({ timeout: 20_000 }); + await expect(panel(page)).toHaveCount(0); + }); + + test("the toggle is keyboard operable and exposes its state", async ({ page }) => { + await mockGallery(page, 12); + await gotoModels(page); + + await toggle(page).focus(); + await expect(toggle(page)).toBeFocused(); + await page.keyboard.press("Enter"); + await expect(toggle(page)).toHaveAttribute("aria-expanded", "true"); + // aria-controls must resolve to the region it actually shows and hides. + await expect(toggle(page)).toHaveAttribute("aria-controls", "rec-models-content"); + await expect(grid(page)).toBeVisible(); + }); + + test("recommendations render and their install buttons still work", async ({ page }) => { + await mockGallery(page, 0); + let installed = null; + await page.route("**/api/models/install/*", (route) => { + installed = decodeURIComponent(new URL(route.request().url()).pathname.split("/").pop()); + return route.fulfill({ contentType: "application/json", body: JSON.stringify({ uuid: "job-1" }) }); + }); + await gotoModels(page); + + const card = grid(page).locator(".rec-models-item", { hasText: "tiny-chat" }); + await expect(card).toBeVisible(); + await expect(card.getByText("512.0 MB")).toBeVisible(); + await card.getByRole("button", { name: "Install" }).click(); + + await expect.poll(() => installed).toBe("tiny-chat"); + }); +}); diff --git a/core/http/react-ui/public/locales/de/models.json b/core/http/react-ui/public/locales/de/models.json index e5929d832e31..3a2f3e387ad0 100644 --- a/core/http/react-ui/public/locales/de/models.json +++ b/core/http/react-ui/public/locales/de/models.json @@ -1,6 +1,17 @@ { "title": "Modelle installieren", "subtitle": "Durchsuchen und installieren Sie KI-Modelle aus der Galerie", + "recommended": { + "title": "Empfohlen für Ihre Hardware", + "cpuNote": "Keine GPU erkannt - kleine Modelle, die auf der CPU reaktionsschnell bleiben.", + "gpuNote": "Passend zum verfügbaren VRAM, mit Spielraum für den Kontext.", + "install": "Installieren", + "installing": "Installation läuft", + "installStarted": "{{model}} wird installiert…", + "installFailed": "Installation fehlgeschlagen: {{message}}", + "dismiss": "Empfehlungen ausblenden", + "summary": "{{n}} Modelle vorgeschlagen" + }, "stats": { "available": "Verfügbar", "installed": "Installiert" @@ -24,8 +35,12 @@ "embedding": "Embedding", "rerank": "Rerank", "fitsGpu": "Passt in die GPU", + "collapseVariants": "Eine Zeile pro Modell", "allBackends": "Alle Backends", - "searchBackends": "Backends suchen..." + "searchBackends": "Backends suchen...", + "contextSize": "Kontext:", + "useCaseLabel": "Nach Anwendungsfall filtern", + "unavailableForBackend": "Für das gewählte Backend nicht verfügbar" }, "search": { "placeholder": "Modelle suchen...", @@ -71,6 +86,7 @@ "empty": { "title": "Keine Modelle gefunden", "withFilters": "Keine Modelle entsprechen den aktuellen Such- oder Filterkriterien.", + "collapsedVariantsHint": "Alternative Builds, die ein anderer Eintrag bereits anbietet, sind ausgeblendet. Schalte \"Eine Zeile pro Modell\" aus, um sie einzubeziehen.", "noFilters": "Die Modellgalerie ist leer." }, "deleteDialog": { @@ -83,5 +99,27 @@ "loadFailed": "Laden der Modelle fehlgeschlagen: {{message}}", "installFailed": "Installation fehlgeschlagen: {{message}}", "deleteFailed": "Löschen fehlgeschlagen: {{message}}" + }, + "variants": { + "title": "Variants", + "chooseVariant": "Choose a variant", + "auto": "Auto", + "autoSelected": "Auto-selected", + "base": "Base build", + "fits": "Fits", + "doesNotFit": "Does not fit", + "unknownSize": "Unknown size", + "unknownBackend": "Unknown backend", + "loading": "Loading variants...", + "quantizationTitle": "Gewichtsformat", + "unknownQuantization": "Format unbekannt", + "showDetails": "Alle Details zu {{variant}} anzeigen", + "hideDetails": "Alle Details zu {{variant}} ausblenden", + "detailsLoading": "Details werden geladen...", + "detailsUnavailable": "Details zu {{variant}} konnten nicht geladen werden.", + "features": { + "dflash": "Schneller: DFlash", + "mtp": "Schneller: MTP" + } } } diff --git a/core/http/react-ui/public/locales/en/models.json b/core/http/react-ui/public/locales/en/models.json index bd23d389ee57..c9a7ab4e9e1a 100644 --- a/core/http/react-ui/public/locales/en/models.json +++ b/core/http/react-ui/public/locales/en/models.json @@ -10,7 +10,8 @@ "installing": "Installing", "installStarted": "Installing {{model}}…", "installFailed": "Install failed: {{message}}", - "dismiss": "Dismiss recommendations" + "dismiss": "Dismiss recommendations", + "summary": "{{n}} models suggested" }, "stats": { "available": "Available", @@ -43,8 +44,12 @@ "vad": "VAD", "ner": "NER", "fitsGpu": "Fits in GPU", + "collapseVariants": "One row per model", "allBackends": "All Backends", - "searchBackends": "Search backends..." + "searchBackends": "Search backends...", + "contextSize": "Context:", + "useCaseLabel": "Filter by use case", + "unavailableForBackend": "Not available for the selected backend" }, "search": { "placeholder": "Search models...", @@ -90,6 +95,7 @@ "empty": { "title": "No models found", "withFilters": "No models match your current search or filters.", + "collapsedVariantsHint": "Alternative builds that another entry already offers are hidden. Turn off \"One row per model\" to include them.", "noFilters": "The model gallery is empty." }, "deleteDialog": { @@ -108,5 +114,27 @@ "selectModel": "Select model...", "searchPlaceholder": "Search models...", "noModels": "No models available" + }, + "variants": { + "title": "Variants", + "chooseVariant": "Choose a variant", + "auto": "Auto", + "autoSelected": "Auto-selected", + "base": "Base build", + "doesNotFit": "Does not fit", + "unknownSize": "Unknown size", + "unknownBackend": "Unknown backend", + "loading": "Loading variants...", + "installVariant": "Install {{variant}}", + "quantizationTitle": "Weight format", + "unknownQuantization": "Unknown format", + "showDetails": "Show full details for {{variant}}", + "hideDetails": "Hide full details for {{variant}}", + "detailsLoading": "Loading details...", + "detailsUnavailable": "Details for {{variant}} could not be loaded.", + "features": { + "dflash": "Faster: DFlash", + "mtp": "Faster: MTP" + } } } diff --git a/core/http/react-ui/public/locales/es/models.json b/core/http/react-ui/public/locales/es/models.json index ac4275fbbbe4..b7329027a49f 100644 --- a/core/http/react-ui/public/locales/es/models.json +++ b/core/http/react-ui/public/locales/es/models.json @@ -1,6 +1,17 @@ { "title": "Instalar modelos", "subtitle": "Explora e instala modelos de IA desde la galería", + "recommended": { + "title": "Recomendado para tu hardware", + "cpuNote": "No se detectó GPU - modelos pequeños que responden bien en CPU.", + "gpuNote": "Ajustados a tu VRAM disponible, con margen para el contexto.", + "install": "Instalar", + "installing": "Instalando", + "installStarted": "Instalando {{model}}…", + "installFailed": "Error al instalar: {{message}}", + "dismiss": "Descartar recomendaciones", + "summary": "{{n}} modelos sugeridos" + }, "stats": { "available": "Disponibles", "installed": "Instalados" @@ -24,8 +35,12 @@ "embedding": "Embedding", "rerank": "Rerank", "fitsGpu": "Cabe en la GPU", + "collapseVariants": "Una fila por modelo", "allBackends": "Todos los backends", - "searchBackends": "Buscar backends..." + "searchBackends": "Buscar backends...", + "contextSize": "Contexto:", + "useCaseLabel": "Filtrar por caso de uso", + "unavailableForBackend": "No disponible para el backend seleccionado" }, "search": { "placeholder": "Buscar modelos...", @@ -71,6 +86,7 @@ "empty": { "title": "No se encontraron modelos", "withFilters": "Ningún modelo coincide con la búsqueda o filtros actuales.", + "collapsedVariantsHint": "Las compilaciones alternativas que otra entrada ya ofrece están ocultas. Desactiva \"Una fila por modelo\" para incluirlas.", "noFilters": "La galería de modelos está vacía." }, "deleteDialog": { @@ -83,5 +99,27 @@ "loadFailed": "Error al cargar modelos: {{message}}", "installFailed": "Error al instalar: {{message}}", "deleteFailed": "Error al eliminar: {{message}}" + }, + "variants": { + "title": "Variants", + "chooseVariant": "Choose a variant", + "auto": "Auto", + "autoSelected": "Auto-selected", + "base": "Base build", + "fits": "Fits", + "doesNotFit": "Does not fit", + "unknownSize": "Unknown size", + "unknownBackend": "Unknown backend", + "loading": "Loading variants...", + "quantizationTitle": "Formato de pesos", + "unknownQuantization": "Formato desconocido", + "showDetails": "Mostrar todos los detalles de {{variant}}", + "hideDetails": "Ocultar todos los detalles de {{variant}}", + "detailsLoading": "Cargando detalles...", + "detailsUnavailable": "No se pudieron cargar los detalles de {{variant}}.", + "features": { + "dflash": "Más rápido: DFlash", + "mtp": "Más rápido: MTP" + } } } diff --git a/core/http/react-ui/public/locales/id/models.json b/core/http/react-ui/public/locales/id/models.json index a8c5404fa43d..87ef33bf8858 100644 --- a/core/http/react-ui/public/locales/id/models.json +++ b/core/http/react-ui/public/locales/id/models.json @@ -2,6 +2,17 @@ "title": "Instal Model", "subtitle": "Telusuri dan instal model AI dari galeri", "models": "Model", + "recommended": { + "title": "Direkomendasikan untuk perangkat keras Anda", + "cpuNote": "GPU tidak terdeteksi - model kecil yang tetap responsif di CPU.", + "gpuNote": "Disesuaikan dengan VRAM yang tersedia, menyisakan ruang untuk konteks.", + "install": "Instal", + "installing": "Menginstal", + "installStarted": "Menginstal {{model}}…", + "installFailed": "Instalasi gagal: {{message}}", + "dismiss": "Tutup rekomendasi", + "summary": "{{n}} model disarankan" + }, "stats": { "available": "Tersedia", "installed": "Terinstal" @@ -31,8 +42,12 @@ "detection": "Deteksi", "vad": "VAD", "fitsGpu": "Muat di GPU", + "collapseVariants": "Satu baris per model", "allBackends": "Semua Backend", - "searchBackends": "Cari backends..." + "searchBackends": "Cari backends...", + "contextSize": "Konteks:", + "useCaseLabel": "Filter berdasarkan kasus penggunaan", + "unavailableForBackend": "Tidak tersedia untuk backend yang dipilih" }, "search": { "placeholder": "Cari model...", @@ -78,6 +93,7 @@ "empty": { "title": "Model tidak ditemukan", "withFilters": "Tidak ada model yang cocok dengan pencarian atau filter Anda.", + "collapsedVariantsHint": "Build alternatif yang sudah ditawarkan entri lain disembunyikan. Matikan \"Satu baris per model\" untuk menyertakannya.", "noFilters": "Galeri model kosong." }, "deleteDialog": { @@ -96,5 +112,27 @@ "selectModel": "Pilih model...", "searchPlaceholder": "Cari model...", "noModels": "Model tidak tersedia" + }, + "variants": { + "title": "Variants", + "chooseVariant": "Choose a variant", + "auto": "Auto", + "autoSelected": "Auto-selected", + "base": "Base build", + "fits": "Fits", + "doesNotFit": "Does not fit", + "unknownSize": "Unknown size", + "unknownBackend": "Unknown backend", + "loading": "Loading variants...", + "quantizationTitle": "Format bobot", + "unknownQuantization": "Format tidak diketahui", + "showDetails": "Tampilkan detail lengkap {{variant}}", + "hideDetails": "Sembunyikan detail lengkap {{variant}}", + "detailsLoading": "Memuat detail...", + "detailsUnavailable": "Detail untuk {{variant}} tidak dapat dimuat.", + "features": { + "dflash": "Lebih cepat: DFlash", + "mtp": "Lebih cepat: MTP" + } } } diff --git a/core/http/react-ui/public/locales/it/models.json b/core/http/react-ui/public/locales/it/models.json index 25c371eb1be6..ee5ad7ff5c74 100644 --- a/core/http/react-ui/public/locales/it/models.json +++ b/core/http/react-ui/public/locales/it/models.json @@ -1,6 +1,17 @@ { "title": "Installa modelli", "subtitle": "Sfoglia e installa modelli AI dalla galleria", + "recommended": { + "title": "Consigliati per il tuo hardware", + "cpuNote": "Nessuna GPU rilevata - modelli piccoli che restano reattivi su CPU.", + "gpuNote": "Dimensionati per la VRAM disponibile, con spazio per il contesto.", + "install": "Installa", + "installing": "Installazione", + "installStarted": "Installazione di {{model}}…", + "installFailed": "Installazione non riuscita: {{message}}", + "dismiss": "Nascondi i consigli", + "summary": "{{n}} modelli suggeriti" + }, "stats": { "available": "Disponibili", "installed": "Installati" @@ -24,8 +35,12 @@ "embedding": "Embedding", "rerank": "Rerank", "fitsGpu": "Entra nella GPU", + "collapseVariants": "Una riga per modello", "allBackends": "Tutti i backend", - "searchBackends": "Cerca backend..." + "searchBackends": "Cerca backend...", + "contextSize": "Contesto:", + "useCaseLabel": "Filtra per caso d'uso", + "unavailableForBackend": "Non disponibile per il backend selezionato" }, "search": { "placeholder": "Cerca modelli...", @@ -71,6 +86,7 @@ "empty": { "title": "Nessun modello trovato", "withFilters": "Nessun modello corrisponde ai filtri attuali.", + "collapsedVariantsHint": "Le build alternative gia offerte da un'altra voce sono nascoste. Disattiva \"Una riga per modello\" per includerle.", "noFilters": "La galleria modelli è vuota." }, "deleteDialog": { @@ -83,5 +99,27 @@ "loadFailed": "Caricamento modelli fallito: {{message}}", "installFailed": "Installazione fallita: {{message}}", "deleteFailed": "Eliminazione fallita: {{message}}" + }, + "variants": { + "title": "Variants", + "chooseVariant": "Choose a variant", + "auto": "Auto", + "autoSelected": "Auto-selected", + "base": "Base build", + "fits": "Fits", + "doesNotFit": "Does not fit", + "unknownSize": "Unknown size", + "unknownBackend": "Unknown backend", + "loading": "Loading variants...", + "quantizationTitle": "Formato dei pesi", + "unknownQuantization": "Formato sconosciuto", + "showDetails": "Mostra tutti i dettagli di {{variant}}", + "hideDetails": "Nascondi tutti i dettagli di {{variant}}", + "detailsLoading": "Caricamento dettagli...", + "detailsUnavailable": "Impossibile caricare i dettagli di {{variant}}.", + "features": { + "dflash": "Più veloce: DFlash", + "mtp": "Più veloce: MTP" + } } } diff --git a/core/http/react-ui/public/locales/ko/models.json b/core/http/react-ui/public/locales/ko/models.json index 2ca05d4a77d9..bb9af8813a05 100644 --- a/core/http/react-ui/public/locales/ko/models.json +++ b/core/http/react-ui/public/locales/ko/models.json @@ -1,6 +1,17 @@ { "title": "모델 설치", "subtitle": "갤러리에서 AI 모델을 둘러보고 설치합니다", + "recommended": { + "title": "하드웨어에 맞는 추천", + "cpuNote": "GPU가 감지되지 않았습니다 - CPU에서도 빠르게 동작하는 소형 모델입니다.", + "gpuNote": "사용 가능한 VRAM에 맞추고 컨텍스트 여유를 남겼습니다.", + "install": "설치", + "installing": "설치 중", + "installStarted": "{{model}} 설치 중…", + "installFailed": "설치 실패: {{message}}", + "dismiss": "추천 닫기", + "summary": "추천 모델 {{n}}개" + }, "stats": { "available": "사용 가능", "installed": "설치됨" @@ -30,8 +41,12 @@ "detection": "감지", "vad": "VAD", "fitsGpu": "GPU에 적합", + "collapseVariants": "모델당 한 행", "allBackends": "모든 백엔드", - "searchBackends": "백엔드 검색..." + "searchBackends": "백엔드 검색...", + "contextSize": "컨텍스트:", + "useCaseLabel": "사용 사례로 필터링", + "unavailableForBackend": "선택한 백엔드에서 사용할 수 없음" }, "search": { "placeholder": "모델 검색...", @@ -77,6 +92,7 @@ "empty": { "title": "모델을 찾을 수 없습니다", "withFilters": "현재 검색 또는 필터와 일치하는 모델이 없습니다.", + "collapsedVariantsHint": "다른 항목이 이미 제공하는 대체 빌드는 숨겨져 있습니다. 포함하려면 \"모델당 한 행\"을 끄세요.", "noFilters": "모델 갤러리가 비어 있습니다." }, "deleteDialog": { diff --git a/core/http/react-ui/public/locales/zh-CN/models.json b/core/http/react-ui/public/locales/zh-CN/models.json index 246153e6894e..f44284757b1f 100644 --- a/core/http/react-ui/public/locales/zh-CN/models.json +++ b/core/http/react-ui/public/locales/zh-CN/models.json @@ -1,6 +1,17 @@ { "title": "安装模型", "subtitle": "从模型库浏览和安装 AI 模型", + "recommended": { + "title": "适合你硬件的推荐", + "cpuNote": "未检测到 GPU - 这些小模型在 CPU 上依然流畅。", + "gpuNote": "已按可用显存选择，并为上下文留出空间。", + "install": "安装", + "installing": "安装中", + "installStarted": "正在安装 {{model}}…", + "installFailed": "安装失败：{{message}}", + "dismiss": "关闭推荐", + "summary": "已推荐 {{n}} 个模型" + }, "stats": { "available": "可用", "installed": "已安装" @@ -24,8 +35,12 @@ "embedding": "嵌入", "rerank": "重排", "fitsGpu": "适合 GPU", + "collapseVariants": "每个模型一行", "allBackends": "所有后端", - "searchBackends": "搜索后端..." + "searchBackends": "搜索后端...", + "contextSize": "上下文：", + "useCaseLabel": "按用例筛选", + "unavailableForBackend": "所选后端不支持" }, "search": { "placeholder": "搜索模型...", @@ -71,6 +86,7 @@ "empty": { "title": "未找到模型", "withFilters": "没有模型与当前搜索或筛选条件匹配。", + "collapsedVariantsHint": "其他条目已经提供的替代构建已被隐藏。关闭“每个模型一行”即可将其包含在内。", "noFilters": "模型库为空。" }, "deleteDialog": { @@ -83,5 +99,27 @@ "loadFailed": "加载模型失败：{{message}}", "installFailed": "安装失败：{{message}}", "deleteFailed": "删除失败：{{message}}" + }, + "variants": { + "title": "Variants", + "chooseVariant": "Choose a variant", + "auto": "Auto", + "autoSelected": "Auto-selected", + "base": "Base build", + "fits": "Fits", + "doesNotFit": "Does not fit", + "unknownSize": "Unknown size", + "unknownBackend": "Unknown backend", + "loading": "Loading variants...", + "quantizationTitle": "权重格式", + "unknownQuantization": "未知格式", + "showDetails": "显示 {{variant}} 的完整详情", + "hideDetails": "隐藏 {{variant}} 的完整详情", + "detailsLoading": "正在加载详情...", + "detailsUnavailable": "无法加载 {{variant}} 的详情。", + "features": { + "dflash": "更快：DFlash", + "mtp": "更快：MTP" + } } } diff --git a/core/http/react-ui/src/App.css b/core/http/react-ui/src/App.css index 130f33084bd5..a973b12940da 100644 --- a/core/http/react-ui/src/App.css +++ b/core/http/react-ui/src/App.css @@ -2191,6 +2191,68 @@ select.input { user-select: none; white-space: nowrap; } +/* Models gallery filter block. Three bands (query / taxonomy / refinements) + stacked by .filter-bar-group. The refinements live in their own band so the + chip row's wrap height can never displace them. */ +.models-filters__query { + flex-wrap: nowrap; + gap: var(--spacing-sm); +} +.models-filters__backend { + flex: 0 0 auto; + min-width: 0; +} +/* .filter-bar carries its own bottom margin for standalone use; inside the + group the parent's gap owns the rhythm. */ +.models-filters > .filter-bar { + margin-bottom: 0; +} +.models-filters__refine { + display: flex; + align-items: center; + flex-wrap: wrap; + gap: var(--spacing-sm) var(--spacing-lg); + padding-top: var(--spacing-sm); + border-top: 1px solid var(--color-border-subtle); +} +/* Both refinements read as one secondary group: same size, same colour and the + same icon-then-label rhythm as .filter-bar-group__toggle. */ +.models-filters__context { + display: flex; + align-items: center; + gap: var(--spacing-sm); + font-size: var(--text-xs); + color: var(--color-text-secondary); +} +.models-filters__context > label { + display: inline-flex; + align-items: center; + gap: var(--spacing-xs); + white-space: nowrap; + cursor: pointer; +} +.models-filters__context input[type='range'] { + width: 140px; + accent-color: var(--color-primary); + cursor: pointer; +} +.models-filters__context-value { + font-weight: var(--font-weight-medium); + min-width: 3em; + font-variant-numeric: tabular-nums; +} + +@media (max-width: 768px) { + /* Search and backend select each take a full row rather than squeezing the + input below a usable width. */ + .models-filters__query { + flex-wrap: wrap; + } + .models-filters__backend { + flex: 1 1 100%; + } +} + .filter-btn__count { display: inline-flex; align-items: center; @@ -2720,6 +2782,40 @@ button.collapsible-header:focus-visible { border-color: var(--color-primary-border); } +/* The global :focus-visible ring is wrapped in :where(), so it carries the + specificity of a bare :focus-visible - the same (0,1,0) as .filter-btn, which + is declared later and therefore won with its resting shadow. Chips were left + with no visible keyboard focus indicator at all; this restates the ring where + it outranks both the resting and the hover shadow. */ +.filter-btn:focus-visible { + box-shadow: 0 0 0 3px var(--color-focus-ring); + border-color: var(--color-primary); +} + +/* A chip is disabled when the selected backend cannot serve that use case. It + keeps its resting colours so the row still reads as a set, dimmed enough to + register as out of play. */ +.filter-btn:disabled { + opacity: 0.4; + cursor: not-allowed; + box-shadow: none; +} +.filter-btn:disabled:hover { + transform: none; + box-shadow: none; + color: var(--color-text-secondary); + border-color: var(--color-border-default); +} + +@media (prefers-reduced-motion: reduce) { + .filter-btn { + transition: none; + } + .filter-btn:hover { + transform: none; + } +} + /* Login page */ .login-page { min-height: 100vh; @@ -2890,6 +2986,16 @@ button.collapsible-header:focus-visible { margin: 0; } +/* Secondary to .empty-state-text: a hint about a default that may be narrowing + the result, so it must not compete with the reason stated above it. */ +.empty-state-hint { + color: var(--color-text-muted); + font-size: var(--text-sm); + line-height: var(--leading-normal); + max-width: 52ch; + margin: 0; +} + /* Opt-in editorial sub-parts — pages can adopt these class names over time */ .empty-state__eyebrow { display: inline-flex; @@ -6449,24 +6555,47 @@ button.collapsible-header:focus-visible { margin-bottom: var(--spacing-md); padding: var(--spacing-md) var(--spacing-lg); } +/* Collapsed keeps the strip and its heading, only dropping the cards, so the + panel is recoverable in place rather than gone. Tighter block padding is + what actually returns vertical space to the gallery. */ +.rec-models--collapsed { + padding-top: var(--spacing-sm); + padding-bottom: var(--spacing-sm); +} .rec-models-head { display: flex; - align-items: flex-start; - justify-content: space-between; - gap: var(--spacing-md); + align-items: center; + gap: var(--spacing-sm) var(--spacing-md); + flex-wrap: wrap; } -.rec-models-title { - display: flex; +.rec-models-toggle { + display: inline-flex; align-items: center; gap: var(--spacing-sm); - flex-wrap: wrap; + background: none; + border: none; + padding: 0; + margin: 0; + font: inherit; + color: var(--color-text-primary); + cursor: pointer; + text-align: left; } -.rec-models-title i { +.rec-models-toggle i { color: var(--color-primary); } +.rec-models-chevron { + font-size: 0.75rem; + transition: transform 150ms ease; +} +.rec-models--collapsed .rec-models-chevron { + transform: rotate(-90deg); +} .rec-models-note { font-size: 0.8125rem; color: var(--color-text-secondary); + flex: 1 1 12rem; + min-width: 0; } .rec-models-dismiss { background: none; @@ -6481,9 +6610,36 @@ button.collapsible-header:focus-visible { } .rec-models-grid { display: grid; - grid-template-columns: repeat(auto-fill, minmax(220px, 1fr)); + /* min() keeps the 220px track from forcing horizontal overflow on narrow + phones, where one column is all that fits anyway. */ + grid-template-columns: repeat(auto-fill, minmax(min(220px, 100%), 1fr)); gap: var(--spacing-sm); margin-top: var(--spacing-md); + /* Reveal animates opacity and transform only; the collapsed state is the + [hidden] attribute, so no height is being animated. */ + animation: rec-models-reveal 180ms ease-out; +} +/* The class sets display:grid, which would otherwise beat the UA [hidden] rule. */ +.rec-models-grid[hidden] { + display: none; +} +@keyframes rec-models-reveal { + from { + opacity: 0; + transform: translateY(-4px); + } + to { + opacity: 1; + transform: none; + } +} +@media (prefers-reduced-motion: reduce) { + .rec-models-grid { + animation: none; + } + .rec-models-chevron { + transition: none; + } } .rec-models-item { display: flex; @@ -9164,3 +9320,328 @@ button.collapsible-header:focus-visible { .model-chip__state { opacity: 0.85; font-style: normal; } .node-filter { margin-bottom: var(--spacing-lg); } .node-detail__metrics { display: flex; gap: var(--spacing-xl); margin: var(--spacing-md) 0 var(--spacing-lg); flex-wrap: wrap; } + +/* Rendered Markdown --------------------------------------------------------- + Gallery descriptions, backend descriptions and voice notes are all + author-supplied Markdown pasted from model cards. Left unscoped they inherit + browser defaults, so a description that opens with `#` renders a 2em heading + inside a 13px surface while a `##` further down is indistinguishable from + body text. This maps the whole element set onto the UI type scale so any + description reads as part of the app rather than as a document dropped into + it. Apply it to every element fed by renderMarkdown(). */ +.markdown-body { + color: var(--color-text-secondary); + font-size: var(--text-sm); + line-height: var(--leading-normal); +} +.markdown-body > :first-child { margin-top: 0; } +.markdown-body > :last-child { margin-bottom: 0; } + +.markdown-body h1, +.markdown-body h2, +.markdown-body h3, +.markdown-body h4, +.markdown-body h5, +.markdown-body h6 { + color: var(--color-text-primary); + font-weight: var(--font-weight-semibold); + line-height: var(--leading-tight); + margin: var(--spacing-md) 0 var(--spacing-xs); +} +.markdown-body h1 { font-size: var(--text-lg); } +.markdown-body h2 { font-size: var(--text-base); } +.markdown-body h3 { font-size: var(--text-sm); } +/* Below h3 the scale has no room left to separate levels by size, so the + remaining depth is carried by case and weight instead. */ +.markdown-body h4, +.markdown-body h5, +.markdown-body h6 { + font-size: var(--text-xs); + color: var(--color-text-secondary); + text-transform: uppercase; + letter-spacing: 0.06em; +} + +.markdown-body p { margin: 0 0 var(--spacing-sm); } +.markdown-body ul, +.markdown-body ol { + margin: 0 0 var(--spacing-sm); + padding-left: var(--spacing-lg); +} +.markdown-body li { margin-bottom: var(--spacing-xs); } +.markdown-body li:last-child { margin-bottom: 0; } +.markdown-body li > ul, +.markdown-body li > ol { margin-top: var(--spacing-xs); } + +.markdown-body a { + color: var(--color-primary); + text-decoration: underline; + text-underline-offset: 2px; +} +.markdown-body a:hover { color: var(--color-primary-hover); } + +.markdown-body code { + font-family: var(--font-mono); + font-size: 0.9em; + background: var(--color-bg-tertiary); + border-radius: var(--radius-sm); + padding: 1px 4px; +} +.markdown-body pre { + margin: 0 0 var(--spacing-sm); + padding: var(--spacing-sm); + background: var(--color-surface-sunken); + border: 1px solid var(--color-border-subtle); + border-radius: var(--radius-md); + overflow-x: auto; +} +.markdown-body pre code { + background: none; + padding: 0; + font-size: var(--text-xs); +} + +.markdown-body blockquote { + margin: 0 0 var(--spacing-sm); + padding-left: var(--spacing-sm); + border-left: 2px solid var(--color-border-strong); + color: var(--color-text-muted); +} +.markdown-body img { + max-width: 100%; + height: auto; + border-radius: var(--radius-sm); +} +.markdown-body hr { + border: 0; + border-top: 1px solid var(--color-border-divider); + margin: var(--spacing-md) 0; +} +/* Tables are the one element that can exceed the measure, so they scroll in + their own box rather than widening the surface around them. */ +.markdown-body table { + display: block; + width: max-content; + max-width: 100%; + overflow-x: auto; + border-collapse: collapse; + margin: 0 0 var(--spacing-sm); + font-size: var(--text-xs); +} +.markdown-body th, +.markdown-body td { + border: 1px solid var(--color-border-subtle); + padding: var(--spacing-xs) var(--spacing-sm); + text-align: left; +} +.markdown-body th { + background: var(--color-bg-tertiary); + font-weight: var(--font-weight-medium); +} + +/* Prose block in a detail pane. The label/value table beside it is right for + scalars, but multi-paragraph prose in a value cell runs the full pane width + and swamps the rows it sits above, so the description gets its own + full-width block with a reading measure. */ +.detail-prose { + margin-bottom: var(--spacing-md); +} +.detail-prose__label { + font-size: var(--text-sm); + font-weight: var(--font-weight-medium); + color: var(--color-text-secondary); + margin-bottom: var(--spacing-xs); +} +.detail-prose__body { + max-width: 68ch; +} + +/* Variant list -------------------------------------------------------------- + One row per alternative build of the same model. Names vary in length, so + the rows share the parent's tracks and line up as columns; the name leads + because it is what the reader is choosing between. Only the informative + status is badged: "fits" is true of nearly every row and says nothing, while + a build that does not fit, and the one that plain Install would pick, do. */ +.variant-list { + display: grid; + width: 100%; + /* info, name, backend, quantization, size, status, action, filler. + Every informative track is content-sized and the filler takes the slack, + so the rows stay packed together rather than stranding the install + affordance an inch of empty space away from the name it acts on. The + grid still spans the pane, which is what lets a revealed detail panel + have the full width its file table needs. */ + grid-template-columns: max-content minmax(0, auto) max-content max-content max-content max-content max-content 1fr; + column-gap: var(--spacing-md); + row-gap: 2px; +} +/* One variant: the info control, the install row, and the details the control + reveals. It spans the list's tracks and passes them down, so the install row + still shares its columns with every other row despite the extra nesting. */ +.variant-entry { + display: grid; + grid-column: 1 / -1; + grid-template-columns: subgrid; + align-items: center; +} +.variant-row { + display: grid; + /* Stops short of the filler track so the hover and focus box hugs the row's + own content rather than trailing across the empty right-hand side. */ + grid-column: 2 / -2; + grid-template-columns: subgrid; + align-items: center; + width: 100%; + margin: 0; + padding: var(--spacing-xs) var(--spacing-sm); + font: inherit; + text-align: left; + color: inherit; + background: none; + border: 1px solid transparent; + border-radius: var(--radius-sm); + cursor: pointer; + transition: background-color var(--duration-fast) var(--ease-default), + border-color var(--duration-fast) var(--ease-default); +} +/* Subgrid keeps the columns shared across rows; without it each row falls back + to sizing its own tracks, which is ragged but still legible. */ +@supports not (grid-template-columns: subgrid) { + .variant-entry { grid-template-columns: max-content minmax(22ch, auto) max-content max-content max-content max-content max-content 1fr; } + .variant-row { grid-template-columns: minmax(22ch, auto) max-content max-content max-content max-content max-content; } +} +.variant-row:hover:not(:disabled) { + background: var(--color-bg-hover); + border-color: var(--color-border-default); +} +.variant-row:focus-visible { + outline: 2px solid var(--color-focus-ring); + outline-offset: 1px; +} +.variant-row:disabled { + cursor: not-allowed; + opacity: 0.55; +} +.variant-row__name { + font-family: var(--font-mono); + font-size: var(--text-sm); + font-weight: var(--font-weight-medium); + color: var(--color-text-primary); + overflow: hidden; + text-overflow: ellipsis; +} +.variant-row__backend, +.variant-row__size { + font-size: var(--text-xs); + color: var(--color-text-muted); +} +/* Monospaced like the name above it, because a weight format is a literal + token and Q4_K_M vs Q4_K_S has to be told apart at a glance in a column. */ +.variant-row__quant { + font-family: var(--font-mono); + font-size: var(--text-xs); + color: var(--color-text-secondary); + white-space: nowrap; +} +/* An entry naming no weight format is a real state, not a gap: it recedes to + the muted colour and drops the mono face, so it reads as prose saying + "unknown" rather than as a token. */ +.variant-row__quant--unknown { + font-family: inherit; + font-style: italic; + color: var(--color-text-disabled); +} +.variant-row__size { + text-align: right; + font-variant-numeric: tabular-nums; +} +.variant-row__status { + display: inline-flex; + align-items: center; + gap: var(--spacing-xs); + flex-wrap: wrap; +} +.variant-row__action { + color: var(--color-text-disabled); + font-size: var(--text-xs); + transition: color var(--duration-fast) var(--ease-default); +} +.variant-row:hover:not(:disabled) .variant-row__action, +.variant-row:focus-visible .variant-row__action { + color: var(--color-primary); +} +/* A build that does not fit stays installable, an explicit choice being an + override the server honours, but it should not read as a peer of the ones + that do. */ +.variant-row--unfit .variant-row__name { + color: var(--color-text-muted); +} + +/* Quiet until asked for: the row's job is to be compared and installed, so the + way into "everything about this one" recedes until the eye or the keyboard + lands on it. */ +.variant-row__info { + grid-column: 1; + display: inline-flex; + align-items: center; + justify-content: center; + width: 1.75rem; + height: 1.75rem; + margin: 0; + padding: 0; + font-size: var(--text-xs); + color: var(--color-text-disabled); + background: none; + border: 1px solid transparent; + border-radius: var(--radius-sm); + cursor: pointer; + transition: color var(--duration-fast) var(--ease-default), + background-color var(--duration-fast) var(--ease-default); +} +.variant-row__info:hover, +.variant-entry:hover .variant-row__info { + color: var(--color-primary); +} +.variant-row__info:hover { + background: var(--color-bg-hover); +} +.variant-row__info:focus-visible { + outline: 2px solid var(--color-focus-ring); + outline-offset: 1px; + color: var(--color-primary); +} +.variant-row__info[aria-expanded="true"] { + color: var(--color-primary); + background: var(--color-bg-hover); + border-color: var(--color-border-default); +} + +/* The third level of disclosure on this page, so it leans on an inset and a + rule rather than another card: a nested panel with its own border inside an + expanded row inside an expanded table reads as debris. */ +.variant-detail { + grid-column: 1 / -1; + min-width: 0; + margin: var(--spacing-xs) 0 var(--spacing-sm) var(--spacing-lg); + border-left: 2px solid var(--color-border-default); + background: var(--color-surface-sunken); + border-radius: 0 var(--radius-md) var(--radius-md) 0; + overflow: hidden; +} +.variant-detail__state { + display: flex; + align-items: center; + gap: var(--spacing-sm); + padding: var(--spacing-sm) var(--spacing-md); + font-size: var(--text-sm); + color: var(--color-text-muted); +} +.variant-detail__state--error { + color: var(--color-error); +} + +@media (prefers-reduced-motion: reduce) { + .variant-row, + .variant-row__info, + .variant-row__action { transition: none; } +} diff --git a/core/http/react-ui/src/components/RecommendedModels.jsx b/core/http/react-ui/src/components/RecommendedModels.jsx index 7620406c869e..55ec4e94468c 100644 --- a/core/http/react-ui/src/components/RecommendedModels.jsx +++ b/core/http/react-ui/src/components/RecommendedModels.jsx @@ -3,28 +3,68 @@ import { useTranslation } from 'react-i18next' import { modelsApi } from '../utils/api' import { useRecommendedModels, isNvfp4Name } from '../hooks/useRecommendedModels' -const DISMISS_KEY = 'localai_rec_models_dismissed' +// Page-scoped storage keys, matching the Models page convention +// (localai-models-*). The underscore key is the pre-rename one: reading it +// keeps an existing dismissal honoured instead of resurrecting the panel for +// everyone who already closed it. +const DISMISS_KEY = 'localai-models-recommended-dismissed' +const LEGACY_DISMISS_KEY = 'localai_rec_models_dismissed' +const COLLAPSE_KEY = 'localai-models-recommended-collapsed' +const CONTENT_ID = 'rec-models-content' + +function readDismissed() { + try { + return localStorage.getItem(DISMISS_KEY) === '1' || localStorage.getItem(LEGACY_DISMISS_KEY) === '1' + } catch { + return false + } +} + +// null means "the user has never chosen", which is what lets the installed +// count pick the default instead of overriding an explicit preference. +function readCollapsePref() { + try { + const raw = localStorage.getItem(COLLAPSE_KEY) + if (raw === '1') return true + if (raw === '0') return false + } catch { /* ignore */ } + return null +} // "Recommended for your hardware" strip at the top of the Models gallery. Shares // the hardware-fit ranking with the empty-state starter widget via -// useRecommendedModels, but styled for the gallery page and dismissible (the -// gallery is a repeat-visit surface, so it shouldn't nag). -export default function RecommendedModels({ addToast }) { +// useRecommendedModels, but styled for the gallery page. +// +// Prominence tracks need: someone with nothing installed is exactly who this is +// for and gets it expanded, while a stocked instance gets a one-line summary +// that still expands on demand. Both the collapse choice and the dismissal +// persist, so the gallery stops re-litigating the decision on every visit. +export default function RecommendedModels({ addToast, installedCount = null }) { const { t } = useTranslation('models') const { recommended, tier, loading } = useRecommendedModels({ count: 4 }) const [installing, setInstalling] = useState(() => new Set()) - const [dismissed, setDismissed] = useState(() => { - try { return localStorage.getItem(DISMISS_KEY) === '1' } catch { return false } - }) + const [dismissed, setDismissed] = useState(readDismissed) + const [collapsePref, setCollapsePref] = useState(readCollapsePref) if (loading || dismissed) return null if (!recommended || recommended.length === 0) return null + // Wait for the installed count before committing to a default: rendering + // expanded and collapsing a frame later would shove the gallery around. + if (installedCount === null || installedCount === undefined) return null + + const collapsed = collapsePref === null ? installedCount > 0 : collapsePref const dismiss = () => { try { localStorage.setItem(DISMISS_KEY, '1') } catch { /* ignore */ } setDismissed(true) } + const toggle = () => { + const next = !collapsed + try { localStorage.setItem(COLLAPSE_KEY, next ? '1' : '0') } catch { /* ignore */ } + setCollapsePref(next) + } + const install = async (name) => { setInstalling(prev => new Set(prev).add(name)) try { @@ -43,18 +83,33 @@ export default function RecommendedModels({ addToast }) { const isGpu = tier.id !== 'cpu' return ( -
+
-
+ {/* The accessible name is the visible title alone; the note sits + outside the control so the name stays short and matches the label a + voice-control user would speak. State comes from aria-expanded. */} +