Request for Gemma 3 4B & Gemma 4 Integration Support with llama-cpp-python

[LeConsulat2]

Hi team,

I would like to request official support and documentation for Gemma 3 4B and Gemma 4 integration within llama-cpp-python.

It would be helpful to have clarification regarding:

• Current compatibility status
• Required llama.cpp / llama-cpp-python versions
• Chat template handling requirements
• Recommended GGUF formats and quantizations
• Known limitations or issues
• Recommended inference parameters for stable usage

Any guidance, implementation notes, or examples would be greatly appreciated (without using ollama)

Kind regards,
Jonathan Park

[Siddhuuuu]

Hey, I looked into this in some depth today and sharing what I found, mapped to your questions above.

Current compatibility status

Confirmed working. I loaded gemma-4-E4B-it-Q4_0.gguf (with mmproj-gemma-4-E4B-it-Q8_0.gguf) through Gemma4ChatHandler and ran an actual chat completion. The model responded coherently, with no template artifacts, and both the vision and audio projectors loaded and warmed up without errors.

> Say hello and count to 5.
Hello!

1
2
3
4
5

Required versions

Tested against vendor/llama.cpp commit e3546c7948e3af463d0b401e6421d5a4c2faf565 and llama_cpp_python==0.3.34 (current main as of today). Both the gemma3/gemma3n and Gemma 4 architectures are present in the vendored source.

Chat template handling

This is the interesting part. Gemma4ChatHandler is currently just pass (it inherits MTMDChatHandler, after the custom Jinja template originally added in #2241 was removed by the #2256 generic-handler refactor), but it still worked correctly.

It looks like MTMDChatHandler pulls the chat template directly from the GGUF's embedded chat_template metadata rather than needing a Python-side override.

One thing I noticed: the rendered prompt used <|turn>user ... <turn|> markers rather than Gemma 3's <start_of_turn>/<end_of_turn>. I'm not sure whether that's Gemma 4's actual new format or something specific to this GGUF, but the model handled it correctly either way.

Recommended GGUF / quantization

I only tested Q4_0 (main model) + Q8_0 (mmproj) on CPU. That combination works cleanly, but I haven't compared other quantization levels yet, so I can't comment on the tradeoffs beyond saying that this configuration works.

Known limitations

The load log surfaces a clear upstream warning that audio input is experimental ("audio input is in experimental stage and may have reduced quality"). Everything I tested was text-only prompting, so I can't personally confirm image or audio input quality yet.

Inference parameters

I only ran with the defaults (n_ctx=2048, n_gpu_layers=0). The GGUF embeds top_k=64, top_p=0.95, and temperature=1.0 as its own defaults. I haven't done enough runs to say anything about stability beyond this one working cleanly.

---

Given this all looks functional, would it be useful if I opened a small docs PR summarizing the confirmed working setup, versions tested, the chat-template source note, and the audio caveat? Happy to put that together if it's welcome as I didn't want to jump straight to a PR without checking first.