LLM Inference Engine

A GPT-2 small (124M) inference engine built from scratch in C++, with no ML frameworks: a byte-level BPE tokenizer, a .safetensors weight loader, multi-head causal attention, a KV cache, temperature/top-k sampling, and int8 quantization, all hand-written and verified against PyTorch/Hugging Face reference outputs at every stage. Runs real GPT-2 weights and generates real text.

$ ./build/llm_engine.exe "The quick brown fox"
The quick brown foxes are a great way to get a little bit of a kick out of your dog.

Results

Correctness	every math kernel, the full transformer block, and the full 12-layer forward pass match PyTorch/HF reference outputs (216k+ test assertions, tests/)
Speed	0.6 → ~19 tok/s on a 4-core/8-thread laptop CPU — a ~32x speedup, from a KV cache (~12x) and AVX2/FMA SIMD + size-gated OpenMP threading (~2.6x)
Memory	int8 weight quantization: 4.0x smaller weight footprint (494 MB → 124 MB), with measured (not assumed) speed and generation-quality tradeoffs — see below

See docs/WRITEUP.md for the full build story, including the two dead-end performance assumptions that measurement caught (naive threading was slower than no threading; quantization's speed win needs real int8 SIMD, not just smaller storage).

What I learned

• Single-token decode is memory-bandwidth-bound, not compute-bound — each weight is read once and used once, which is why the KV cache (~12x, eliminating recomputation) mattered far more than SIMD/threading (~2.6x, speeding up compute that wasn't the real bottleneck).
• "More threads" isn't free. Naively threading every matmul was slower than no threading — spawn/join overhead beat the tiny per-head attention matmuls. Fixed by gating threading on work size, but only because I benchmarked with threading off first.
• A memory win and a speed win are different claims. int8 quantization gave the full theoretical 4x memory reduction but barely moved speed, because the inner loop still multiplies in float — real throughput needs int8 SIMD, not just a smaller representation.
• Greedy decoding amplifies tiny numeric differences. One flipped argmax early in generation cascades into a fully different continuation, so token-level exact-match is a poor quality metric under any numerical perturbation (quantization, different BLAS, fp16) even when output stays fluent.
• Read the actual file, not the paper. "124M parameters" undercounts what's really in the checkpoint by ~13M — a per-layer causal-mask buffer that's stored but not learned.

Build

Requires CMake 3.20+, a C++20 compiler (MSYS2/MinGW-w64 g++ on Windows), OpenMP, and nlohmann-json + doctest:

pacman -S mingw-w64-x86_64-nlohmann-json mingw-w64-x86_64-doctest

Needs GPT-2 small's weights/tokenizer downloaded to models/gpt2/ first (safetensors + vocab.json + merges.txt + config.json from openai-community/gpt2 on Hugging Face — not committed, see .gitignore).

cmake -S . -B build -G Ninja
cmake --build build

./build/llm_engine.exe "your prompt here"
./build/llm_engine_tests.exe    # run from repo root (loads tests/fixtures/*.json)
./build/bench_quantize.exe      # float32 vs int8: memory, speed, quality

Test fixtures are generated from scripts/gen_*_reference.py (needs numpy/torch) and checked in under tests/fixtures/ — no Python needed just to build and test.

How it's built

• Tokenizer (bpe_tokenizer.*, utf8.hpp) — GPT-2's byte-level BPE from scratch: byte↔unicode remapping, hand-rolled Unicode-aware pretokenization (C++'s std::regex has no \p{L}/\p{N} support), and rank-ordered greedy merging. Exact match vs HF's tokenizer, including accented characters, code indentation, and multi-space runs.
• Weight loading (safetensors.*, gpt2_config.*) — parses the .safetensors header/offset format directly; byte-exact vs PyTorch.
• Math kernels (ops.*) — matmul, softmax, layernorm, GELU (tanh approximation), all verified against numpy/PyTorch to several decimal places.
• Transformer (attention.*, mlp.*, gpt2_block.*, gpt2_model.*) — causal multi-head attention and the feed-forward block, composed into the full 12-layer model with weight-tied output projection. Matches HF's real GPT2Block/GPT2LMHeadModel output on real weights.
• KV cache (kv_cache.hpp) — incremental per-token attention that reuses cached K/V instead of recomputing the whole sequence; bit-identical output to the uncached path, ~12x faster.
• Sampling (sampling.*) — greedy argmax, or temperature + top-k sampling with a verified empirical distribution.
• Performance (ops.cpp) — AVX2/FMA SIMD intrinsics on the matmul inner loop, plus OpenMP threading gated by a work-size threshold (see Results above for why the gate matters).
• Quantization (quantize.*, quantized_model.*) — int8 weights with per-tensor symmetric scale, a standalone bench_quantize report comparing float32 vs int8 memory/speed/quality on real weights.

Roadmap

• Phase 0 — toolchain, repo, build
• Phase 1 — load weights & tokenizer
• Phase 2 — math building blocks (matmul, softmax, layernorm, GELU)
• Phase 3 — one transformer layer (attention + feed-forward)
• Phase 4 — full forward pass, first generated text
• Phase 5 — KV cache, temperature/top-k sampling
• Phase 6 — performance (SIMD, cache-aware matmul, threading)
• Phase 7 — quantization (stretch)
• Phase 8 — polish & benchmarks

Detailed benchmarks

Greedy-decoding 20 tokens from the prompt "The quick brown fox" on an Intel i7-8565U (4C/8T), single sequence, GPT-2 small (124M):

Stage	tok/s	vs. previous
Phase 4 — naive (recompute full sequence every step)	0.6	—
Phase 5 — + KV cache	7.4	~12x
Phase 6 — + AVX2/FMA SIMD matmul	11.6	~1.6x
Phase 6 — + OpenMP threading (size-gated)	~19	~1.65x

Total: ~32x over the Phase 4 baseline.

The threading number needed a size gate to actually help: naively parallelizing every matmul (including attention's tiny per-head ones, K=64) made things slower than SIMD alone (10.4 tok/s vs 11.6) — OpenMP's thread-spawn/join cost exceeded the work being parallelized. Threading only pays off once a matmul's total work (M*K*N) clears a threshold; below that it runs single-threaded SIMD. Measured, not assumed — see src/ops.cpp.

Phase 7 — int8 quantization

./build/bench_quantize.exe quantizes every big per-layer matmul weight (wte, c_attn, c_proj, c_fc, mlp.c_proj) to int8 with one float scale per tensor, and compares against the float32 model on the same prompt/token count:

	float32	int8
weight memory (quantized tensors)	494 MB	124 MB (4.0x smaller, exactly as expected)
speed	17.1 tok/s	17.8 tok/s (~4% faster)
generated text	"...a great way to get a little bit of a kick..."	"...a common sight in the wild, but they are..."

Two honest findings, not just a headline number:

• The memory win is exactly the theoretical 4x (int8 vs float32), but speed barely moved. The dequantize-then-FMA inner loop (src/quantize.cpp) still does the multiply-accumulate in float — it reads a quarter as many bytes per weight, but real throughput needs actual int8 SIMD (AVX-VNNI / _mm256_dpbusd_epi32 or similar), which this implementation doesn't do. So: real memory savings, close to zero speed savings, without that next step.
• Generated text diverges from the float baseline within a few tokens (3/20 generated tokens matched by position). This isn't quantization being "wrong" — greedy decoding is inherently sensitive to tiny logit perturbations: one flipped argmax early on feeds back into every subsequent token, so any numerical difference (quantization, a different BLAS, even fp16 vs fp32) tends to cascade into a fully different continuation. Both outputs are fluent, on-topic English; the interesting comparison would be perplexity/likelihood of the reference continuation under the quantized model, not exact-match, which isn't implemented here.