Author: João Felipe De Souza
Empirical profiling of quantization tradeoffs in LLM inference: throughput, latency, memory, and perplexity across FP32, FP16, INT8, and INT4.
For detailed architecture and methodology, see DESIGN.md.
Quantization is widely used in production inference, but its benefit depends on:
- GPU architecture
- model size
- batch size
- kernel implementation
The common assumption is:
“lower precision = faster inference”
This project shows that this is not always true.
- GPT-2 (117M)
- GPT-2-medium (345M)
- FP32
- FP16
- INT8
- INT4 (NF4)
- NVIDIA RTX 2070 (8.6 GB VRAM)
- CUDA 13.0
- bitsandbytes 0.49.2
- batch sizes: 1, 2, 4, 8, 16
- sequence lengths: 128, 512
- decode steps: 32
Total:
- 2 models
- 4 quantization levels
- 2 sequence lengths
- 5 batch sizes
- 80 measured runs
| Quantization | Peak throughput | Model memory |
|---|---|---|
| FP32 | 2621 tok/s | 0.464 GB |
| FP16 | 3313 tok/s | 0.232 GB |
| INT4 | 1313 tok/s | 0.113 GB |
| INT8 | 488 tok/s | 0.153 GB |
| Quantization | Peak throughput | Model memory |
|---|---|---|
| FP32 | 1096 tok/s | 1.322 GB |
| FP16 | 1673 tok/s | 0.661 GB |
| INT4 | 639 tok/s | 0.239 GB |
| INT8 | 232 tok/s | 0.380 GB |
FP16 delivers:
- 1.26× speedup over FP32 on GPT-2
- 1.53× speedup over FP32 on GPT-2-medium
- 50% lower model memory
| Model | FP32 | FP16 | INT8 | INT4 |
|---|---|---|---|---|
| GPT-2 | 1.7998 | 1.7991 | 1.8032 | 1.8023 |
| GPT-2-medium | 1.7610 | 1.7619 | 1.7625 | 1.7593 |
Perplexity changes are extremely small.
| Quantization | Model memory | Savings vs FP32 |
|---|---|---|
| FP32 | 1.322 GB | — |
| FP16 | 0.661 GB | 50.0% |
| INT8 | 0.380 GB | 71.3% |
| INT4 | 0.239 GB | 81.9% |
INT4 compresses GPT-2-medium to ~18% of FP32 memory.
The surprising result:
- INT8 is much slower than FP16 and FP32
- INT4 is better than INT8, but still slower than FP16
- This holds across both GPT-2 and GPT-2-medium
This suggests that on consumer Turing GPUs, bitsandbytes quantization kernels are not throughput-optimal for small-to-medium GPT-2 models.
Quantization does not improve single-GPU decode throughput on this hardware. Its value is:
- fitting larger models in memory
- reducing VRAM footprint
- potentially enabling larger serving batches
If the goal is minimum memory with minimal quality loss:
- INT4 is clearly best
If the goal is maximum throughput:
- FP16 is clearly best
On an RTX 2070 / Turing-class GPU:
- FP16 is the best performance setting
- INT4 is the best memory-saving setting
- INT8 is not attractive in this workload
- Quantization preserves perplexity, but does not automatically improve speed
This is a strong reminder that quantization must be evaluated on the target hardware — not assumed to help universally.
| File | Description |
|---|---|
| results/quantization_results.csv | Full 80-run sweep |
| results/quantization_summary.csv | Speedup / memory savings vs FP32 |
| results/metadata.json | Run configuration |
| File | Description |
|---|---|
| plots/gpt2_throughput_vs_batch.png | GPT-2 throughput scaling |
| plots/gpt2-medium_throughput_vs_batch.png | GPT-2-medium throughput scaling |
| plots/quantization_summary.png | Memory, throughput, perplexity |
| plots/quantization_tradeoffs.png | Speedup, memory savings, perplexity delta |
quantization-profiler/
├── quantization_profiler.py
├── plot_quantization.py
├── README.md
├── DESIGN.md
├── LICENSE
├── requirements.txt
├── results/
└── plots/
python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt
python3 quantization_profiler.py
python3 plot_quantization.py
- Only GPT-2 family models tested
- Only RTX 2070 hardware tested
- bitsandbytes kernel behavior may differ on Ampere/Hopper GPUs
- Batch sweep capped at 16 for runtime practicality
- No downstream task accuracy, only perplexity
- Dettmers et al., LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022)
- Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (2023)