Gemma x vLLM: Live Inference Demo

A small, self-contained web app that shows, live and side by side, how much faster vLLM serves a model compared to plain HuggingFace Transformers.

Type a question, watch both engines stream their answer at the same time (vLLM visibly races ahead), then run a load test to see how vLLM pulls even further ahead when many requests arrive at once.

Model: google/gemma-3-270m-it (Gemma 3, 270M, instruction-tuned), small enough to run on a modest GPU.

---

What you get

Section	What it does
Ask, stream both	Sends your prompt to HuggingFace and vLLM in parallel and streams both answers token by token into two boxes. Shows tokens/sec for each and the single-request speed-up.
Run load test	Fires N concurrent requests at each engine and compares throughput and total time. Produces the two charts shown below.
Model and HF token	Set or replace your Hugging Face token and (re)load the model without touching files.

---

Environment we tested on

These results were measured on a single laptop GPU under WSL2:

Component	Value
GPU	NVIDIA GeForce RTX 4060 Laptop GPU (8 GB, compute capability 8.9)
OS	Windows + WSL2 (kernel 6.18 microsoft-standard-WSL2)
Python	3.12.3
PyTorch	2.11.0 + cu130
Transformers	5.13.0
vLLM	0.24.0
Gradio	6.19.0
Model / precision	google/gemma-3-270m-it, bfloat16
vLLM settings	max_model_len 1024, gpu_memory_utilization 0.30, enforce_eager

Note: the GPU is shared with the Windows desktop, so only about 2.5 GB is really allocatable. The demo is tuned to stay under that ceiling (bf16, low memory utilization, eager mode).

---

Results

Single request (streaming)

Both engines answer "Explain what a large language model is in simple terms." at the same time. vLLM finishes first.

Engine	Tokens	Time	Speed
HuggingFace	100	4.05 s	24.7 tok/s
vLLM	100	2.77 s	36.1 tok/s

Even for a single request, vLLM is about 1.5× faster. Its real advantage, however, comes under concurrent load, where continuous batching allows it to serve many requests simultaneously.

Load test: 8 concurrent requests

This is where vLLM really wins. HuggingFace handles requests one at a time; vLLM batches all 8 together.

Metric	HuggingFace	vLLM	Speed-up
Total time	14.7 s	2.3 s	6.4x faster
Throughput	34.9 tok/s	218.9 tok/s	6.3x

Throughput: how much work gets done

Throughput is how many words (tokens) the system produces every second, counting all users together. Higher is better. With 8 people asking at once, vLLM produces about 6x more text per second than HuggingFace on the exact same GPU and model.

Total time: how long everyone waits

This is how long it took to finish all 8 answers. Lower is better. HuggingFace takes longer because it answers one person at a time; vLLM serves all 8 together, so everyone gets their answer much sooner.

Throughput scaling: what happens as more users arrive

Here we go from 1 user to 4 to 8. HuggingFace stays flat: it can only do one request at a time, so extra users don't help, they just wait. vLLM climbs because it batches requests together, doing more total work as load grows. The wider the gap, the more users a single GPU can serve.

p95 latency: the wait your slowest users feel

"p95 latency" means 95 out of 100 requests finish faster than this line, it's the wait time for your slowest users, which is what people actually complain about. Lower is better. As load rises, HuggingFace's wait balloons because requests pile up in a queue, while vLLM stays low and steady, everyone still gets a quick reply.

---

Setup

1. Environment

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

On a WSL2 / CUDA machine, install torch and vllm that match your CUDA build first (see the vLLM install docs), then run pip install -r requirements.txt for the rest.

2. Hugging Face access (Gemma is a gated model, one time)

1. Open https://huggingface.co/google/gemma-3-270m-it and click Agree and access repository.
2. Create a Read token at https://huggingface.co/settings/tokens.
3. Provide the token in either way:
   • Copy .env.example to .env and paste it after HF_TOKEN=, or
   • Paste it into the HF token box in the app and click Load / reload model.

3. Run

python app.py

Open http://localhost:7860. The model is loaded and warmed up at startup, so the first query is already fast.

---

Configuration (.env)

app.py auto-loads a .env file next to it (gitignored, never committed). Copy .env.example to .env and edit:

Key	Default	Purpose
HF_TOKEN	(blank)	Hugging Face token for the gated Gemma model
DEMO_MODEL	google/gemma-3-270m-it	Model to serve (any HF causal LM works)
DEMO_DTYPE	bfloat16	Precision. Gemma 3 requires bf16
DEMO_MAX_MODEL_LEN	1024	vLLM max context length
DEMO_GPU_MEM_UTIL	0.30	Fraction of GPU memory vLLM may use
DEMO_PORT	7860	Web UI port

Real shell environment variables override .env. To try it without Gemma access, point it at an ungated model:

DEMO_MODEL=HuggingFaceTB/SmolLM-135M DEMO_DTYPE=float16 python app.py

---

Understanding the plots

After you click Run load test you get two bar charts, HuggingFace (red) vs vLLM (green):

• Throughput (tokens / second), higher is better. Total tokens produced across all N requests divided by the wall-clock time. vLLM's bar is taller because it batches many requests together instead of handling them one at a time.

• Total time (seconds), lower is better. How long it took to finish all N requests. vLLM's bar is much shorter.

The table above the charts also prints the speed-up (vLLM throughput divided by HuggingFace throughput). Raise the Concurrent requests slider and run again: the gap grows, because that is exactly where vLLM's continuous batching and PagedAttention pay off. A single request shows a smaller gap; concurrent load shows the real story.

---

How it works

• HuggingFace runs the model the naive way, one request at a time. That is the baseline most people start with.
• vLLM serves the same weights with an optimized engine: continuous batching (it does not wait for a batch to fill) and PagedAttention (efficient GPU memory for the KV cache). Same answers, far more throughput under load.

Both engines are loaded once at startup and kept in memory. vLLM is driven through its async streaming API so tokens appear as they are generated.

---

Notes

• Tuned to run on a memory-constrained WSL2 GPU (bf16, low gpu_memory_utilization, enforce_eager). If you hit an out-of-memory error, lower DEMO_GPU_MEM_UTIL.
• Everything is one file (app.py) plus config, easy to read and adapt.