diff --git a/docs/content/features/model-gallery.md b/docs/content/features/model-gallery.md index 56dde2118220..a1a2b2ee3573 100644 --- a/docs/content/features/model-gallery.md +++ b/docs/content/features/model-gallery.md @@ -61,6 +61,21 @@ Select a specific build with `--variant minicpm-v-4_7-1b-q4`, `--variant minicpm-v-4_7-1b-q5`, `--variant minicpm-v-4_7-1b-q6`, or `--variant minicpm-v-4_7-1b-q8`. +### Underdog Saluki abliterated builds + +`underdog-saluki-27b-1.0-abliterated` provides Kujira's mixed IQ2 adaptation +of Underdog Saluki for text chat, reasoning, and tool use. Its MTP variant +enables multi-token prediction with up to two draft tokens. Both builds use +the same weights, the embedded chat template, and a 32,768-token context. + +To select the MTP build explicitly: + +```bash +local-ai models install --variant underdog-saluki-27b-1.0-abliterated-mtp underdog-saluki-27b-1.0-abliterated +``` + +These entries are text-only and do not include a vision projector. + ### Repairing an existing model configuration Gallery template updates do not rewrite YAML files for already installed models. diff --git a/gallery/index.yaml b/gallery/index.yaml index 981cc3563d0d..384077559eea 100644 --- a/gallery/index.yaml +++ b/gallery/index.yaml @@ -3,6 +3,93 @@ # written by scripts/models/cmd/modelcatalog and stays only so releases before # the cutover keep working. CI fails when it differs from the generated copy: # go run ./scripts/models/cmd/modelcatalog --library gallery/models/index.yaml --out gallery/index.yaml +- url: github:mudler/LocalAI/gallery/virtual.yaml@master + name: underdog-saluki-27b-1.0-abliterated + description: | + Kujira's abliterated adaptation of Underdog Saluki 27B 1.0 uses mixed + IQ2 weights with an embedded MTP head. This text-only llama.cpp build + uses ordinary decoding; the MTP variant enables speculative decoding. + license: apache-2.0 + urls: + - https://huggingface.co/ConwayResearch/Underdog-Saluki-27B-1.0 + - https://huggingface.co/Kujira/Underdog-Saluki-27B-1.0-MTP-Abliterated-GGUF + tags: + - llm + - gguf + - cpu + - gpu + - reasoning + - function-calling + - uncensored + files: + - filename: llama-cpp/models/underdog-saluki-27b-1.0-abliterated/Underdog-Saluki-27B-1.0-IQ2-mix-MTP-abliterated.gguf + sha256: b9b231748c94e13ecc0f841942798eb36b27eaea60014fea896218bc86742bbc + uri: https://huggingface.co/Kujira/Underdog-Saluki-27B-1.0-MTP-Abliterated-GGUF/resolve/c1d826d73831efcb819d9d64c625f0fab56ae8fc/Underdog-Saluki-27B-1.0-IQ2-mix-MTP-abliterated.gguf + overrides: + backend: llama-cpp + context_size: 32768 + function: + automatic_tool_parsing_fallback: true + grammar: + disable: true + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/underdog-saluki-27b-1.0-abliterated/Underdog-Saluki-27B-1.0-IQ2-mix-MTP-abliterated.gguf + presence_penalty: 1.1 + temperature: 1.0 + top_k: 20 + top_p: 0.95 + template: + use_tokenizer_template: true + variants: + - model: underdog-saluki-27b-1.0-abliterated-mtp +- url: github:mudler/LocalAI/gallery/virtual.yaml@master + name: underdog-saluki-27b-1.0-abliterated-mtp + description: | + Kujira's abliterated adaptation of Underdog Saluki 27B 1.0 uses mixed + IQ2 weights with an embedded MTP head. This text-only llama.cpp build + enables multi-token prediction with up to two draft tokens. + license: apache-2.0 + urls: + - https://huggingface.co/ConwayResearch/Underdog-Saluki-27B-1.0 + - https://huggingface.co/Kujira/Underdog-Saluki-27B-1.0-MTP-Abliterated-GGUF + tags: + - llm + - gguf + - cpu + - gpu + - reasoning + - function-calling + - uncensored + - mtp + files: + - filename: llama-cpp/models/underdog-saluki-27b-1.0-abliterated/Underdog-Saluki-27B-1.0-IQ2-mix-MTP-abliterated.gguf + sha256: b9b231748c94e13ecc0f841942798eb36b27eaea60014fea896218bc86742bbc + uri: https://huggingface.co/Kujira/Underdog-Saluki-27B-1.0-MTP-Abliterated-GGUF/resolve/c1d826d73831efcb819d9d64c625f0fab56ae8fc/Underdog-Saluki-27B-1.0-IQ2-mix-MTP-abliterated.gguf + overrides: + backend: llama-cpp + context_size: 32768 + function: + automatic_tool_parsing_fallback: true + grammar: + disable: true + known_usecases: + - chat + options: + - use_jinja:true + - spec_type:draft-mtp + - spec_n_max:2 + parameters: + model: llama-cpp/models/underdog-saluki-27b-1.0-abliterated/Underdog-Saluki-27B-1.0-IQ2-mix-MTP-abliterated.gguf + presence_penalty: 1.1 + temperature: 1.0 + top_k: 20 + top_p: 0.95 + template: + use_tokenizer_template: true - url: github:mudler/LocalAI/gallery/virtual.yaml@master name: adavar-3b-q4 description: | diff --git a/gallery/models/index.yaml b/gallery/models/index.yaml index da124ab3a6ca..9d51e10d29ad 100644 --- a/gallery/models/index.yaml +++ b/gallery/models/index.yaml @@ -4,6 +4,109 @@ # changes and never edit it by hand. CI fails when it is stale. A build from # master reads it as the default model source; a tagged build reads the gallery # until the release cutover. +- kind: model + id: underdog-saluki-27b-1.0-abliterated + name: underdog-saluki-27b-1.0-abliterated + version: 0.0.0-gallery.cdab61f388f4 + description: | + Kujira's abliterated adaptation of Underdog Saluki 27B 1.0 uses mixed + IQ2 weights with an embedded MTP head. This text-only llama.cpp build + uses ordinary decoding; the MTP variant enables speculative decoding. + license: apache-2.0 + urls: + - https://huggingface.co/ConwayResearch/Underdog-Saluki-27B-1.0 + - https://huggingface.co/Kujira/Underdog-Saluki-27B-1.0-MTP-Abliterated-GGUF + tags: + - llm + - gguf + - cpu + - gpu + - reasoning + - function-calling + - uncensored + channel: gallery + policy: + requireSignature: false + schemaVersion: 1 + model: + url: github:mudler/LocalAI/gallery/virtual.yaml@master + files: + - filename: llama-cpp/models/underdog-saluki-27b-1.0-abliterated/Underdog-Saluki-27B-1.0-IQ2-mix-MTP-abliterated.gguf + sha256: b9b231748c94e13ecc0f841942798eb36b27eaea60014fea896218bc86742bbc + uri: https://huggingface.co/Kujira/Underdog-Saluki-27B-1.0-MTP-Abliterated-GGUF/resolve/c1d826d73831efcb819d9d64c625f0fab56ae8fc/Underdog-Saluki-27B-1.0-IQ2-mix-MTP-abliterated.gguf + overrides: + backend: llama-cpp + context_size: 32768 + function: + automatic_tool_parsing_fallback: true + grammar: + disable: true + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/underdog-saluki-27b-1.0-abliterated/Underdog-Saluki-27B-1.0-IQ2-mix-MTP-abliterated.gguf + presence_penalty: 1.1 + temperature: 1.0 + top_k: 20 + top_p: 0.95 + template: + use_tokenizer_template: true + variants: + - model: underdog-saluki-27b-1.0-abliterated-mtp +- kind: model + id: underdog-saluki-27b-1.0-abliterated-mtp + name: underdog-saluki-27b-1.0-abliterated-mtp + version: 0.0.0-gallery.774b2b47f0dc + description: | + Kujira's abliterated adaptation of Underdog Saluki 27B 1.0 uses mixed + IQ2 weights with an embedded MTP head. This text-only llama.cpp build + enables multi-token prediction with up to two draft tokens. + license: apache-2.0 + urls: + - https://huggingface.co/ConwayResearch/Underdog-Saluki-27B-1.0 + - https://huggingface.co/Kujira/Underdog-Saluki-27B-1.0-MTP-Abliterated-GGUF + tags: + - llm + - gguf + - cpu + - gpu + - reasoning + - function-calling + - uncensored + - mtp + channel: gallery + policy: + requireSignature: false + schemaVersion: 1 + model: + url: github:mudler/LocalAI/gallery/virtual.yaml@master + files: + - filename: llama-cpp/models/underdog-saluki-27b-1.0-abliterated/Underdog-Saluki-27B-1.0-IQ2-mix-MTP-abliterated.gguf + sha256: b9b231748c94e13ecc0f841942798eb36b27eaea60014fea896218bc86742bbc + uri: https://huggingface.co/Kujira/Underdog-Saluki-27B-1.0-MTP-Abliterated-GGUF/resolve/c1d826d73831efcb819d9d64c625f0fab56ae8fc/Underdog-Saluki-27B-1.0-IQ2-mix-MTP-abliterated.gguf + overrides: + backend: llama-cpp + context_size: 32768 + function: + automatic_tool_parsing_fallback: true + grammar: + disable: true + known_usecases: + - chat + options: + - use_jinja:true + - spec_type:draft-mtp + - spec_n_max:2 + parameters: + model: llama-cpp/models/underdog-saluki-27b-1.0-abliterated/Underdog-Saluki-27B-1.0-IQ2-mix-MTP-abliterated.gguf + presence_penalty: 1.1 + temperature: 1.0 + top_k: 20 + top_p: 0.95 + template: + use_tokenizer_template: true - kind: model id: adavar-3b-q4 name: adavar-3b-q4