fix(gguf): set ignore_merges for llama 3 gguf tokenizers

[jason-fxz]

Summary

Llama 3 GGUF files (tokenizer.ggml.pre = llama-bpe, llama3, llama-v3) tokenized some words differently from their HF tokenizer and from llama.cpp. Both take a word that is one vocabulary token as is, without running the BPE merges (ignore_merges in the HF tokenizer.json, the same flag in llama.cpp). transformers' GGUF converter never sets it. Follow-up to #722.

Changes

• models/gguf/tokenizer.py: sets ignore_merges on BPE vocabularies with a Llama 3 pre-tokenizer. Example: " việc" is one token (100769) in HF and llama.cpp, but was three (Ġvi, á»ĩ, c).

Results

Every normal vocabulary token plus 20k-60k random concatenations of 2-4 tokens:

File	Compared with	Strings	Mismatches on main	Mismatches with this PR
Llama-3.2-1B-Instruct Q8_0	HF tokenizer.json	146,683	844	0
Llama-3.2-1B-Instruct Q8_0	llama.cpp f2cc728	186,648	1305	0
DeepSeek-R1-Distill-Llama-8B	llama.cpp f2cc728	186,648	–	0

Other pre-tokenizers do not change. tests/tokenizer passes.