Ahead-of-time compiler that turns a quantized GGUF model into one native executable, or into one C library (--lib): no runtime, no Python, no interpreter, no model file to load. The weights are placed in a read-only section of the binary, the OS maps them lazily, and, with the weights already in the page cache, the first forward pass starts about 0.2 ms after main() (0.6–0.7 s to the first token from a cold cache, like any mmap-based loader). Llama, Qwen2/2.5, Qwen3, Phi-3 and Gemma 3 models are supported; --model hf://org/repo:Q4_K_M fetches the GGUF from the Hugging Face Hub. Ready-made binaries for popular small models are published alongside the releases (see Prebuilt model binaries).
$ aot-llm compile --model tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf --output ./tinyllama_bin
[1/4] parsed tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf (GGUF v3, 201 tensors, 23 metadata keys)
tinyllama | llama | 1.100B params | MOSTLY_Q4_K_M | dim 2048 hidden 5632 layers 22 heads 32/4 vocab 32000
[2/4] tokenizer: spm 32000 tokens, chat template: zephyr, blob 613.4 KB
[3/4] generated project in ./tinyllama_bin.build (22 layers unrolled, weights embedded)
[4/4] cargo build --release ...
done: ./tinyllama_bin (642.3 MB) in 10.8s
$ ./tinyllama_bin --prompt "The capital of France is" -n 32
Paris.
2. B.C. The capital of ancient Rome was Rome.
[aot-llm] startup 0.190 ms (threads 0.111, tokenize 0.076) | prompt 6 tok (97 tok/s) | gen 32 tok (107 tok/s) | peak RSS 608 MB | 8 threads | neon+dotprod
The output is byte-for-byte identical to llama.cpp (Ollama 0.34.4) for the same GGUF under greedy decoding; see Correctness for what that does and does not cover.
Requirements: Rust 1.98+ (the kernels use vdotq_s32), a GGUF of a supported architecture: Llama (TinyLlama, Llama 2/3/3.2, Mistral 7B, ...), Qwen2/2.5, Qwen3, Phi-3 (4k-context variants) or Gemma 3. --model also takes hf://org/repo:QUANT (or hf://org/repo/file.gguf): the file is downloaded once into ~/.cache/aot-llm (HF_TOKEN for gated repositories).
git clone https://github.com/p10node/aot-llm && cd aot-llm
cargo build --release
# Inspect what will be compiled
./target/release/aot-llm inspect ./Llama-3.2-1B-Instruct-Q4_K_M.gguf
# Compile (about 10-15 s on an M1 Pro; the GGUF is embedded, not copied)
./target/release/aot-llm compile --model ./Llama-3.2-1B-Instruct-Q4_K_M.gguf --output ./llama32
# Optional: bake the KV cache of a fixed system prompt into the binary (two build passes)
./target/release/aot-llm compile --model ./Llama-3.2-1B-Instruct-Q4_K_M.gguf --output ./llama32 \
--system "You are a concise assistant. Answer in one short paragraph."
# Optional: also compile the Metal GPU path (macOS, Apple Silicon); the binary then uses the GPU by default
./target/release/aot-llm compile --model ./Llama-3.2-1B-Instruct-Q4_K_M.gguf --output ./llama32_gpu --gpu metal
./llama32_gpu --prompt "The capital of France is" -n 32 # --device cpu switches back
# Run
./llama32 --prompt "The capital of France is" -n 32
./llama32 --chat --prompt "Write one sentence about the moon." --temp 0.7
./llama32 --chat --system "You are a concise assistant. Answer in one short paragraph." --prompt "Why is the sky blue?" # uses the baked prefix
./llama32 --info
# Qwen3 (answers directly by default; --think lets it reason first), straight from the Hub
./target/release/aot-llm compile --model hf://unsloth/Qwen3-0.6B-GGUF:Q4_K_M --output ./qwen3 --gpu metal
./qwen3 --chat --prompt "Write one sentence about the moon."
./target/release/aot-llm compile --model hf://unsloth/gemma-3-1b-it-GGUF:Q4_K_M --output ./gemma1b
# A C library instead of an executable (see examples/c/main.c, examples/python/)
./target/release/aot-llm compile --model ./Qwen3-0.6B-Q4_K_M.gguf --output ./libqwen3 --lib --name qwen3
cc -O2 examples/c/main.c -I./libqwen3 ./libqwen3/libqwen3.a -o chat && ./chat "Why is the sky blue?"
# Tools: constrained answers, stop strings, conversations across processes
./qwen3 --chat --prompt "Is this review positive? 'Broke after two days.'" --choices positive,negative,neutral
git diff | ./qwen3 --chat --system "One-line commit subject." --stop '\n'
./qwen3 --chat --prompt "My name is Pierre." --save-kv talk.kv
./qwen3 --chat --prompt "What is my name?" --load-kv talk.kv --save-kv talk.kv # evaluates only the new turnGenerated binary options:
-p, --prompt TEXT Prompt text (reads stdin when omitted)
-n, --max-tokens N Maximum tokens to generate [default: 128]
--temp F Sampling temperature, 0 = greedy [default: 0]
--top-k N Top-k filter, 0 = off [default: 40]
--top-p F Nucleus sampling threshold [default: 0.95]
--repeat-penalty F Repetition penalty, 1 = off [default: 1.0]
--seed N RNG seed [default: 42]
-c, --ctx N Context length [default: prompt + max-tokens]
-t, --threads N Worker threads [default: min(cores, 8)]
-b, --batch N Prompt tokens per batched step, 1 = token by token [default: 64]
--chat Wrap the prompt in the model's chat template (llama3 / zephyr / chatml / qwen3 / llama2 / mistral / gemma)
--system TEXT System prompt for --chat
--think Let a Qwen3 model think before answering (--chat)
--stop TEXT Stop when the output contains TEXT (repeatable; \n escapes allowed)
--choices A,B,C Answer with exactly one of the given strings (also --choice TEXT, repeatable)
--save-kv FILE After generating, write the tokens and KV cache of the whole run to FILE
--load-kv FILE Continue a saved run: the prompt is appended (--chat: as the next user turn)
--no-bos Do not prepend the BOS token
--no-prefetch Do not page in the weights from background threads at startup
--no-prefix-cache Ignore the compile-time KV prefix cache
--kernels NAME auto | scalar | neon | neon-dotprod | avx2
--device NAME auto | cpu | gpu (gpu needs a binary built with --gpu) [default: auto]
--weights PATH Sidecar weights file (sidecar builds only)
--show-special Print special tokens
--tokens Print token ids instead of text
--dump-logits Print the logits of the last prompt token and exit
-q, --quiet Do not print statistics to stderr
--info Print model information and exit
Compiler options (aot-llm compile --help): --model PATH|hf://org/repo[:QUANT|/file.gguf], --lib [static|shared] (build a model library with the C API instead of an executable; --output is then a directory), --symbol-prefix PFX (prefix the library's C symbols and header names so several models can be linked into one program), --gpu metal (compile the Metal backend in, macOS only), --system TEXT / --prefix TEXT (bake a KV prefix cache), --target <triple>, --native (-C target-cpu=native), --sidecar, --emit-only (generate the project without building), --build-dir, --name, -j, -q.
Debugging: AOT_DEBUG_COMPARE=1 ./bin ... re-evaluates the prompt token by token and reports the first KV-cache divergence from the batched path; --dump-logits prints the logits of the last prompt token.
aot-llm is about first-response latency and packaging, not raw throughput. Same GGUF, same Apple M1 Pro, 8 threads, Ollama 0.34.4 unless noted (details in docs/benchmarks/). Each row compares like with like: warm against warm, cold against cold, first token against first token. (An earlier version of this table set aot-llm's first token against Ollama's time for a whole 64-token request and its warm process start against Ollama's cold model load; both comparisons were wrong and were replaced on 2026-11-03, see docs/benchmarks/08-ollama-0.40-recheck.md.)
| aot-llm | Ollama / llama.cpp | |
|---|---|---|
| warm, per call (weights in the page cache / model resident) | 0.2–0.8 ms process start; nothing stays resident between calls | ~3 ms per HTTP request (0.40.13), with a server holding the model in RAM |
| chat reply with a fixed system prompt, first token, warm | 44–67 ms (CPU), 42–89 ms (Metal, incl. start-up) with the baked KV prefix on Qwen3-0.6B; 90–160 / 64–74 ms on the Llama models | 41–53 ms (CPU and Metal) with the model resident and the system prompt already in the server's KV cache (0.40.13, Qwen3-0.6B): a tie |
| chat reply with a fixed system prompt, first token, cold | 0.6–0.7 s (page-in; the baked prefix still skips the system prompt) | 1.2–2.8 s (model load, then the full prompt) |
| cold (nothing cached), first token | 0.6–0.7 s (paging 640–780 MB of weights in) | 0.5–2.8 s model load, then prompt evaluation; 1.3–4.6 s for a whole 64-token request |
| resident memory | 614–773 MiB (KV cache sized to the request) | 766–1586 MiB at Ollama's default context length, plus ~40 MiB of server/app |
| decode, CPU | 113–128 tok/s | 86–117 tok/s (same machine, measured on different days) |
decode, GPU (--gpu metal) |
92–106 tok/s | 139–150 tok/s (Metal) |
| long prompt evaluation | 200–260 tok/s (CPU), 550–750 tok/s (Metal) | 410–510 tok/s (CPU), 340–390 tok/s (Metal) |
| warm, per call, GPU | ~25–30 ms (Metal device, kernels from the OS cache, weights mapped) | ~3 ms per request once loaded; 0.5–0.8 s model load when the model is not resident |
| deployment | one file, no runtime, no daemon; copy and run; static Linux cross-build | install Ollama, pull the model, keep a server up |
| models | llama, qwen2, qwen3, phi3, gemma3 architectures, 9 tensor types |
every architecture and quantization |
Good fit: command-line tools, agents that call a model many times for short answers, edge / serverless / cron jobs where a process spawns, answers and exits, embedding a model in an application (--lib gives a static or shared library with a C API; see examples/).
Poor fit: multi-user chat servers, very long prompts (RAG, documents) on the CPU, machines whose GPU is not an Apple one (only Metal is supported). vLLM and friends solve a different problem (batched GPU serving, cold starts of tens of seconds) and are not comparable for one process answering one request. llamafile solves the packaging half of this (one file with the weights appended, every llama.cpp architecture, CUDA) but still parses the GGUF and builds the compute graph at start-up, so its start-up cost is llama.cpp's; it has not been benchmarked here.
aot-llm compile takes a GGUF file and builds a program for that one model. It reads the architecture from the GGUF metadata, writes out the forward pass layer by layer with a kernel chosen for each tensor's quantization type, and compiles the result with LTO into a stripped static binary that depends only on std. There is no graph interpreter and nothing to load at run time: the tensor data sits in a read-only section of the executable (via .incbin) and the OS maps it lazily like any other segment.
A few things follow from that, and a few more were added on top:
- It runs on any CPU of its architecture. NEON and AVX2 kernels (plus a scalar fallback) for Q4_0, Q5_0, Q8_0, Q4_K, Q5_K, Q6_K, F16, BF16 and F32, picked once at startup. Prompt tokens go through in batches of up to 64, which makes prompt evaluation several times faster than decoding without changing a bit of the output.
- A fixed system prompt costs nothing per call.
--system "..."evaluates it at build time and bakes its KV cache into the binary; a chat that starts with it skips straight to the user turn. - Conversations survive across processes.
--save-kvwrites the state of a run and--load-kvpicks it up, so a multi-turn dialogue driven by separate invocations only ever evaluates the new turn. - Output can be constrained for tools.
--stop TEXTends generation before a string;--choices a,b,cmasks the logits so the answer is exactly one of the given strings. Classifiers and routers never need to parse anything. - It can be a library instead.
--libproduces a static or shared library plusaot_llm.h, weights embedded the same way, and everything above is in the C API too. See Library mode. - It can use the GPU on Apple Silicon.
--gpu metaladds a Metal path that reads the weights straight from the executable's mapping, no copy, and shares the activations and KV cache with the CPU. The frameworks aredlopened, so the binary still links onlylibSystemand falls back to the CPU when there is no GPU.
Cross-compiling to x86_64-unknown-linux-musl from macOS gives a static ELF, and --sidecar keeps the weights in a file next to the binary if a 600 MB executable is unwelcome. The details are in How it works.
sequenceDiagram
autonumber
participant U as User
participant C as aot-llm (compiler)
participant G as aot-gguf
participant E as aot-codegen
participant R as cargo / rustc
participant B as model binary
participant OS as OS loader
U->>C: aot-llm compile --model m.gguf --output bin
C->>G: mmap m.gguf, parse header + metadata + tensor table
G-->>C: hyper-parameters, tensor offsets / shapes / quant types
C->>E: ModelSpec (validated llama graph)
E->>E: serialise tokenizer blob (sorted index, merges, byte table)
E->>E: emit model.rs (tensor consts + unrolled forward), payload.rs (.incbin), main.rs, rt/*
E->>R: cargo build --release (LTO, codegen-units=1, panic=abort, strip)
R-->>C: standalone executable (weights in a read-only section)
C-->>U: bin
U->>B: ./bin --prompt "..."
OS->>B: map segments (weights not read, only mapped)
B->>B: tokenizer over embedded blob, spawn worker pool, allocate KV cache (~0.2 ms)
loop each token
B->>B: forward(): embed -> [rmsnorm -> q/k/v matvec -> RoPE -> attention -> o matvec -> residual -> rmsnorm -> gate/up -> SwiGLU -> down -> residual] x L -> rmsnorm -> logits
B->>B: sample, decode piece
B-->>U: stream text
end
The static pipeline emitted for every model is
tokens -> embedding -> [RMSNorm -> Attention(GQA, RoPE, KV cache) -> + -> RMSNorm -> SwiGLU FFN -> +] x N -> RMSNorm -> logits -> sampler
Qwen2 adds bias vectors to the q / k / v projections and Qwen3 a per-head RMSNorm of q and k before RoPE; both rotate (i, i + rot_dim / 2) pairs (the GPT-NeoX layout) instead of adjacent elements. Phi-3 stores q / k / v as one attn_qkv tensor and gate / up as one ffn_up tensor; the compiler addresses the parts as row ranges of the same bytes, so the generated code is the plain block. Gemma 3 scales the embedding by sqrt(dim), normalises q and k per head, uses GeGLU, normalises the attention and FFN outputs before the residual adds, and attends over a sliding window of 512 / 1024 positions in five of every six layers with a separate RoPE base (10000, unscaled) while the sixth layer sees the whole context (base 1e6, linear scaling on the 4B+ models); the compiler emits the window and the RoPE table per layer. The code generator emits exactly the operators the architecture has, on the CPU and on the GPU (where the NEOX models rotate in a separate dispatch because the pairs are half a head apart, and the biases are added in the epilogue of the matrix kernels).
Prompt batches run the matrix products over dynamically scheduled row blocks and the element-wise work (RMSNorm, SwiGLU, residual adds, activation quantization) over the pool as well, with the same per-element arithmetic as the token-by-token path, so batched and single evaluation stay bit-identical.
Per matmul, the f32 activation vector is quantized once to Q8_0 (for Q4_0/Q8_0 weights) or Q8_K (for K-quants) and every output row is an integer dot product scaled back to f32, the same scheme ggml uses, so results match llama.cpp. During decoding the q/k/v projections form one parallel region and gate/up another; rows are handed out in dynamic chunks; attention is parallel over heads. The thread pool keeps workers hot between the few hundred small parallel regions of a decode step.
With --gpu metal the generated model.rs also contains forward_gpu / forward_batch_gpu, which encode the same operator sequence into one Metal command buffer per step through rt/metal.rs:
- The weights section is placed in
__DATA(page aligned) and re-mappedMAP_SHAREDfrom the executable file at startup, then wrapped withnewBufferWithBytesNoCopy. A private mapping would be copied page by page the first time the GPU wires it (hundreds of milliseconds), and wired copies of__TEXTpages make the CPU fault on them afterwards; the shared mapping has neither problem and shares the page cache with the loader's own mapping. Mapping 636 MB costs about 15–25 ms on first use. - The activation buffers and the KV cache of
Stateare page-aligned anonymous mappings wrapped the same way, so the CPU embedding lookup, the GPU transformer blocks, the compile-time KV prefix and the sampler all work on one copy of the data. - A decode step is six dispatches per layer: RMSNorm, the
q,k,vprojections in one dispatch (the simdgroups are spread over the three matrices; RoPE applied in the epilogue ofqandk,kandvwritten straight into the cache), attention (one threadgroup per head, online softmax over blocks of 32 positions, optional sliding window),wo(residual add fused), RMSNorm, the fusedgate/up/SwiGLU (or GeGLU) product anddown(residual add fused). The matrix-vector kernels are bound by their vector loads, so RMSNorm is kept as its own 12 µs kernel: folding it into the products (a second vector read per simdgroup) measured 2–3x slower. Prompt batches use a 32x32x32-tile GEMM on the simdgroup matrix units with the weights dequantized once per tile. - The Objective-C runtime and
Metal.frameworkare resolved withdlopen/dlsymon first use; nothing is linked, and--device cpunever loads them. The kernel source (shaders.metal) is embedded and compiled by the system Metal compiler the first time a binary runs on a machine (~0.3 s), then served from the OS shader cache (<1 ms).
The GPU kernels use f32 activations and dequantize weights on the fly (as ggml's Metal path does) instead of the CPU path's Q8 activations, so GPU output is not bit-identical to the CPU path or to llama.cpp's CPU output; it is closer to exact f32 arithmetic (kernel error ~1e-6 relative against an f64 reference versus ~1e-3 for the Q8 path), and greedy generations agree in practice.
Prompt tokens are processed by forward_batch in chunks: every weight row is loaded once per chunk and the 1x4 micro-kernels decode each quantized block once for four activations, so prompt evaluation runs several times faster than decoding. With --system, the compiler builds the binary twice: pass 1 evaluates the system prefix with the final kernels and writes its KV cache, pass 2 embeds that blob in a read-only section; at run time a prompt whose tokens start with the baked prefix loads the rows with a memcpy and evaluates only the user turn.
| crate | role |
|---|---|
crates/aot-gguf |
GGUF v2/v3 parser (header, metadata, tensor descriptors), memory-mapped access, in-memory writer for tests |
crates/aot-kernels |
the runtime: quant layouts, SIMD matvec kernels, RMSNorm / RoPE / attention, thread pool, tokenizer, sampler, CLI driver, C API (capi.rs), Metal backend (metal.rs, shaders.metal). Copied verbatim into every generated project as src/rt/ |
crates/aot-codegen |
graph extraction (ModelSpec), tokenizer blob builder, Rust project emitter (binary or library), the C header (aot_llm.h), cargo driver |
crates/aot-llm |
the aot-llm CLI: compile, inspect, tokenize |
examples/ |
cookbook/ (classify, ask, summarize, extract, multi-turn chat, Python batch labelling), c/main.c (the C API), python/ (ctypes binding), commit-msg/ (a git commit-message generator in one binary) |
scripts/ |
bench.py (benchmark harness), release-binaries.sh (build the prebuilt model binaries and upload them to the Hub) |
scripts/release-binaries.sh build compiles a set of popular small models (Qwen3 0.6B / 1.7B, Qwen2.5 1.5B, Llama 3.2 1B / 3B, Gemma 3 1B, Phi-3 mini) for Apple Silicon (with the Metal backend) and as static x86_64 Linux executables; scripts/release-binaries.sh upload ORG/REPO publishes them to a Hugging Face model repository with a model card. Each file is one executable with the model inside: download, chmod +x, run --chat --prompt "...". The binaries carry the licenses of their models.
examples/cookbook/ holds small finished tools, each a few lines around one compiled binary: a classifier that answers with exactly one label (--choices), a natural-language-to-shell helper that stops at the first newline (--stop), a clipboard summarizer, a field extractor emitting one JSON object per line, a multi-turn chat whose turns are separate processes (--save-kv / --load-kv), and a Python batch labeller on the C library.
aot-llm compile --lib builds the same generated project as a staticlib (--lib shared: cdylib) whose entry points are the functions of aot_llm.h:
aot_model *m = aot_open(NULL); /* weights mapped, tokenizer ready, threads up: ~1 ms warm */
aot_ctx *c = aot_ctx_new(m, 512, 0); /* one sequence, 512-token context */
aot_generate_params p = aot_generate_default_params();
p.chat = 1; p.system = "You are a concise assistant."; p.max_tokens = 96;
aot_generate(c, "Why is the sky blue?", 0, &p, print_piece, NULL); /* streams pieces to the callback */
uint32_t ids[64];
int32_t n = aot_tokenize(m, "The capital of France is", 0, -1, 1, ids, 64);
aot_ctx_reset(c);
aot_eval(c, ids, n, 1);
const float *logits = aot_logits(c); /* n_vocab floats */The model is a process-wide singleton (the library is the model); contexts and samplers are plain handles. Evaluations are serialised internally, so contexts can be driven from any thread. aot_generate appends to its context, so calling it again continues the conversation; aot_ctx_save / aot_ctx_load move a conversation between processes; aot_generate_params.stop and .choices are the CLI's --stop / --choices. A compile-time KV prefix (--system) is honoured by aot_eval when a fresh context starts with the baked tokens. --gpu metal works the same way as for executables (aot_model_params.device). Link with cc app.c libmodel.a on macOS and cc app.c libmodel.a -lpthread -ldl -lm on Linux; the compiler prints the line. To link several models into one program build each with --symbol-prefix qwen3_ (and so on): the symbols and the header's identifiers get the prefix.
examples/c/main.c is a complete program, examples/python/aot_llm.py a ctypes binding with the same surface (Model.generate, streaming, save / load, choices, stop), and examples/commit-msg/ shows the executable side: a commit-message generator with the instructions baked in, usable as a prepare-commit-msg hook.
The tables below are the initial measurements of the first release. Every later optimisation step is measured with scripts/bench.py and recorded under docs/benchmarks/ (one file per step, cold/warm/chat columns, plus the Ollama reference). Summary of the steps so far (Apple M1 Pro, 8 threads, medians; "chat" = 88-token prompt with a 66-token system prompt for TinyLlama, 70/55 for Llama 3.2):
| step | TinyLlama Q4_K_M chat first token | Llama-3.2-1B chat first token | TinyLlama decode | Llama-3.2-1B decode |
|---|---|---|---|---|
baseline (bf6f95b) |
685 ms | 777 ms | 104 tok/s | 61–87 tok/s |
| batched prompt + 1x4 micro-kernels | 424 ms | 488 ms | 116 tok/s | 83 tok/s |
| + background prefault | 375 ms (cold first token 652 → 642 ms) | 302 ms (1030 → 711 ms) | 114 tok/s | 93 tok/s |
+ fused q/k/v, dynamic chunks, fcvt scales |
436 ms | 298 ms | 128 tok/s | 113 tok/s |
+ baked system-prompt KV cache (--system) |
164 ms | 90 ms | 128 tok/s | 113 tok/s |
Metal backend (--gpu metal), same prompts on the GPU |
64 ms (+30 ms Metal start-up) | 74 ms (+30 ms) | 106 tok/s | 92 tok/s |
Decode is now at llama.cpp's CPU speed (117 / 86 tok/s in Ollama), and a chat reply with a baked system prompt starts in about 100 ms instead of the 700–1000 ms of the first release (aot-llm's own baseline; a warm Ollama server with the same system prompt in its KV cache answers in about 50 ms too, see the table above). The Metal path trades ~30 ms of start-up (loading the frameworks, creating the device, kernels from the OS shader cache, wiring the weights) for prompt evaluation at 550–750 tok/s on 300-token prompts (CPU: 200–260), chat first tokens of 64–74 ms and decoding at 92–106 tok/s on these 1B models, slightly below the CPU path on TinyLlama and a little above it on Llama 3.2 (the GPU kernels reach 75–107 GB/s against llama.cpp's ~115 GB/s). It is the better choice for long prompts and for models the CPU cannot keep up with; details in docs/benchmarks/07-metal.md. The baseline tables that follow are kept for reference.
Machine: Apple M1 Pro (8 performance + 2 efficiency cores, 16 GB), macOS, Rust 1.99. Prompt: 11-13 tokens, 64 generated tokens, greedy, warm page cache, 8 threads, median of 3 runs on an otherwise idle machine. Ollama 0.34.4 with its bundled llama.cpp llama-server, same GGUF files imported with ollama create, measured through /api/generate (raw: true); "cold" means after ollama stop.
| TinyLlama-1.1B Q4_K_M | Llama-3.2-1B-Instruct Q4_K_M | |
|---|---|---|
| aot-llm process start -> ready to decode | 0.19 ms (threads 0.11, tokenize 0.08) | 0.27 ms |
| aot-llm time to first token (warm cache) | 120 ms (prompt eval) | 110 ms |
| Ollama cold request, CPU (model load / total) | 1.57 s / 2.33 s | 2.84 s / 4.61 s |
| Ollama cold request, Metal GPU (model load / total) | 0.54 s / 1.34 s | 0.83 s / 1.48 s |
| Ollama warm request, total for 64 tokens (model already resident) | ~0.58 s (the per-request overhead itself is a few ms, ~3 ms with 0.40.13; the rest is prompt evaluation and decoding) | ~0.78 s |
"Ready" for aot-llm means the weights are mapped, the tokenizer has encoded the prompt, the worker pool is up and the KV cache is allocated. Nothing is read from disk at startup; the first forward pass pages the weights in (about 0.6 s on a cold page cache for 640 MB, like any mmap-based loader).
| TinyLlama-1.1B Q4_K_M | Llama-3.2-1B-Instruct Q4_K_M | |
|---|---|---|
| aot-llm peak RSS (whole process, incl. KV cache) | 608 MiB | 770 MiB |
| aot-llm binary on disk | 642 MiB | 776 MiB |
Ollama llama-server RSS, CPU mode |
798 MiB | 1586 MiB |
Ollama llama-server RSS, GPU mode |
766 MiB | 1420 MiB |
Ollama ollama serve + app processes |
~40 MiB | ~40 MiB |
aot-llm's RSS is essentially the touched weight pages (636 / 763 MiB of tensor data) plus a context-sized KV cache; there is no second copy of anything. Ollama's figures are at its default context length, while aot-llm sizes the KV cache to the request (--ctx), so the two are not like for like at long contexts.
| TinyLlama-1.1B Q4_K_M | TinyLlama-1.1B Q4_0 | Llama-3.2-1B Q4_K_M | |
|---|---|---|---|
| aot-llm decode | 107 tok/s | 85 tok/s | 86 tok/s |
| aot-llm prompt eval (token by token) | 107 tok/s | 90 tok/s | 100 tok/s |
| Ollama / llama.cpp CPU decode | 117 tok/s | - | 86 tok/s |
| Ollama / llama.cpp CPU prompt eval (batched) | 510 tok/s | - | 412 tok/s |
| Ollama / llama.cpp Metal GPU decode | 150 tok/s | - | 139 tok/s |
Decode speed is within ~10% of llama.cpp's CPU path. Prompt evaluation is the known gap: aot-llm processes prompt tokens one at a time (matrix-vector), while llama.cpp batches them into matrix-matrix products. Using 10 threads on this machine (i.e. the two efficiency cores) drops throughput to ~20 tok/s because every parallel region waits for its slowest worker, hence the default cap of 8.
vLLM was not benchmarked: it needs a CUDA GPU, and it is a serving engine (continuous batching, paged attention) whose cold start loads weights into GPU memory and captures CUDA graphs, which takes seconds to minutes. It is the wrong tool for the "start a process, answer, exit" use case this project targets, which is why the comparison above is against Ollama.
cargo test --workspaceruns- unit tests for the GGUF parser, f16 conversion, quantizers, pool, ops, sampler and tokenizer pre-tokenizers;
crates/aot-kernels/tests/dots.rs: every SIMD dot product (NEON, NEON+dotprod, AVX2, AVX2+F16C) against the scalar reference and a float reference, for every supported weight type;crates/aot-kernels/tests/metal.rs(macOS): every Metal kernel against the CPU operators: the matrix product for all nine weight formats (single vector, batched, partial tiles, residual accumulation), the fused RoPE / gate / RMSNorm variants, RMSNorm, RoPE, attention for 64- and 128-wide heads, SwiGLU; skipped when no Metal device exists;crates/aot-codegen/tests/tokenizer_blob.rs: SentencePiece and byte-level BPE round trips through the serialised blob;crates/aot-llm/tests/e2e.rs: builds a tiny random GGUF for each architecture (llama,qwen2with biases,qwen3with q/k norms andq_dim > dim,phi3with fused tensors) that uses all nine weight types, compiles it with the CLI, runs the binary with--dump-logitsand compares against an f32 reference forward pass (max error 0.0000 on the logits); on macOS it also compiles each model with--gpu metaland checks the GPU logits (single-token and batched prompt paths) against the unquantized reference. Further tests compile the model with--lib, build a C client againstaot_llm.hwith the system C compiler and check tokens, logits (batched, token by token, and afteraot_ctx_save/aot_ctx_load),aot_generateandchoices; and exercise--stop,--choicesand the--save-kv/--load-kvcontinuation (bit-identical to evaluating the whole prompt at once).
- The same suite runs in CI on Linux x86_64 (AVX2) and macOS Apple Silicon (NEON + Metal), plus a musl cross-build (
.github/workflows/ci.yml). - Tokenization was checked against llama.cpp ids for the three vocabularies (e.g. Llama 3
"Hello world"->[128000, 9906, 1917], TinyLlama ->[1, 15043, 3186]; Qwen uses the Llama 3 regex with single-digit numbers). - Greedy generations of the real models matched Ollama 0.34.4 (its bundled llama.cpp) token for token when validated in October 2026: TinyLlama-1.1B, Llama-3.2-1B, Qwen2.5-0.5B-Instruct (Q8_0 and Q4_K_M, the latter mostly
q5_0) on CPU and Metal, Qwen3-0.6B on CPU and Metal for raw prompts (the chat prompt of this 0.6B model is a low-margin case where llama.cpp's own CPU and Metal backends disagree with each other as well), Phi-3-mini-4k-instruct Q4_K_M (38-token chat reply identical to llama.cpp's CPU output; its Metal output differs from both, as llama.cpp's own does), Gemma-3-1b-it and Gemma-3-4b-it Q4_K_M (raw prompts identical to llama.cpp's CPU output; the chat reply of the 1B model is identical to llama.cpp's Metal output and differs from its CPU output, whose GELU goes through ggml's f16 lookup table while aot-llm evaluates the tanh form exactly). Re-checked 2026-11-03 against Ollama 0.40.13: the Qwen3-0.6B raw-prompt CPU output now matches llama.cpp's Metal output and differs from its CPU output (llama.cpp's two backends also differ from each other on that prompt); llama.cpp's ARM CPU kernels changed between the two versions. "Identical to llama.cpp" therefore means identical to a given llama.cpp build on prompts where the top two logits are not within rounding of each other; it is not a stable property across llama.cpp releases. - The AVX2 kernels were validated by running the test suite under
docker run --platform linux/amd64; a TinyLlama binary cross-compiled tox86_64-unknown-linux-muslruns in the same container (avx2+f16c).
rustup target add x86_64-unknown-linux-musl
aot-llm compile --model m.gguf --output ./m_linux --target x86_64-unknown-linux-musl
file ./m_linux # ELF 64-bit LSB pie executable, x86-64, static-pie linked, strippedFor *-musl targets the generated project links with rust-lld and the self-contained musl CRT, so no cross toolchain is needed. Other targets need a linker for that platform (.cargo/config.toml in the generated project is the place to set it).
- Architectures as written by
convert_hf_to_gguf.py:llama: Llama 1/2/3/3.1/3.2, TinyLlama, Mistral 7B, Vicuna, OpenLLaMA and other derivatives without biases. RoPE "normal" layout, grouped-query attention, RMSNorm, SwiGLU, tied or separate output head, Llama 3.x RoPE frequency factors, linear RoPE scaling.qwen2: Qwen2 / Qwen2.5 (q/k/v biases, NEOX RoPE). The 0.5B models havedim = 896, so theirQ4_K_Mfiles are mostlyq5_0, which is supported.qwen3: Qwen3 dense models (per-head q/k RMSNorm, NEOX RoPE, explicit head size).--chatanswers without thinking by default;--thinkkeeps the thinking turn.phi3: Phi-3 mini / medium with the plain 4k context (fusedattn_qkv/ffn_uptensors, NEOX RoPE,<|user|>/<|end|>chat template). The 128k variants use LongRoPE (rope_factors_long/short) and are rejected; Phi-4-mini's tokenizer (gpt-4opre-tokenizer) is not implemented.gemma3: Gemma 3 1B / 4B / 12B / 27B text models (scaled embedding, q/k norms, post-norms, GeGLU, sliding-window layers with their own RoPE base, head size 256). The vision tower of the multimodal GGUFs is a separate file and is ignored; the sliding window keeps a full-size KV cache (no ring buffer), so memory grows with the context like a global layer's.
- Tensor types:
f32,f16,bf16,q4_0,q5_0,q8_0,q4_K,q5_K,q6_K(coversQ4_0,Q5_0,Q8_0,Q4_K_S/M,Q5_K_S/M,Q6_K,F16files).iq*,q2_K,q3_K,q4_1,q5_1are rejected with a message naming the tensor. - Tokenizers: SentencePiece (
llama) and byte-level BPE (gpt2with the Llama 3, Qwen2 or GPT-2 pre-tokenizer). Chat templates: llama3, zephyr, chatml, qwen3, phi3, llama2, mistral, gemma. - Single sequence, batch size 1, unix targets (the weight embedding uses
global_asm!, memory mapping usesmmap). GPU: Metal on macOS / Apple Silicon only (--gpu metal), head sizes of 32, 64, 96 or 128. - Not yet: KV-cache quantization, Phi-4 / Gemma 2 / MoE architectures, LongRoPE, CUDA / Vulkan backends.
unsafe is confined to SIMD intrinsics, the .incbin symbol and mmap views, the fork-join pool's lifetime-erased job pointer and disjoint-range writes, the Metal backend's objc_msgSend calls and the raw handles of the C API; each site is commented with its invariant.
MIT or Apache-2.0, at your option.