Basalt is a local inference engine for mixture-of-experts models: Qwen3.8-Flash-Next and DeepSeek-V4.1-Flash. It runs on an NVIDIA Blackwell GPU (sm_120) on Linux and Windows x64, with the experts that do not fit in VRAM in system RAM (and optionally on more GPUs). The server speaks the OpenAI and Anthropic APIs. Website: basalt.jgulfo.com.

Help wanted: spare Blackwell compute for testing? See Contributing.

Requirements

• Linux x86-64 with AVX2, or Windows 10 1903+ x64 (platforms).
• An NVIDIA Blackwell GPU (compute capability 12.0) with CUDA 13.
• CMake 3.24+, GCC 13+ (Linux), Python 3.10+, and enough system RAM for the experts that do not fit in VRAM.

Quick start

Prebuilt Linux and Windows archives are attached to each release (what they need); otherwise build from source:

git clone https://github.com/jesdga95/basalt && cd basalt
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j                  # bin/basalt, bin/basalt-bench, bin/basalt-probe, bin/basalt-vision
pip install -r requirements.txt
hf download jesdga/Qwen3.8-Flash-Next-Basalt Qwen3.8-FN-NVFP4-DQ6_K.basalt --local-dir models

Put the settings in basalt_config.json:

{
  "version": 2,
  "models": [{ "model": "models/Qwen3.8-FN-NVFP4-DQ6_K.basalt", "gpu": "0,1", "kv": "int8" }],
  "engine": { "resident-experts": true },
  "server": { "port": 8080 }
}

bin/basalt serve --config basalt_config.json

The server listens on 127.0.0.1. Every setting: configuration. More examples: example configurations.

Models

	Qwen3.8-Flash-Next	DeepSeek-V4.1-Flash
Packs	default Qwen3.8-FN-NVFP4-DQ6_K, Hugging Face	default DeepSeek-V4.1-Flash-1017-UD-IQ1_M, Hugging Face
Context	default 8,192; --max-context sets it	default 8,192; --max-context sets it, up to 1,048,576
Drafting	MTP, on by default	DSpark, on by default
Images	yes	yes
Documents (PDF, text)	yes	yes
Video	yes, with FFmpeg (video)	no
Guide	guide	guide
Precision	precision	precision

All models compared: models.

Documentation

• WSL2 deployment notes: this fork's setup, native model storage, locked memory and Codex.

• Run: running, configuration, models and the router, sampling, image sources, documents, diagnostics, basalt-probe, platforms.

• Models: models, Flash-Next, DeepSeek, the .basalt file, file format changes.

• Quality: precision, method.

• Build and develop: building, adding a model, community runs, changelog, contributing.

Credits

• Strata: the project Basalt forks (MIT).
• Qwen and DeepSeek: the models. Their packs carry the models' own licences (see the Hugging Face cards).
• unsloth (GGUFs), NVIDIA (NVFP4 experts).
• @giveen: the BF16 reference for the DeepSeek quality numbers.
• ggml and llama.cpp: quantization formats and kernels in src/ (MIT, THIRD_PARTY_LICENSES).

License

MIT, see LICENSE.