A Python/Cython implementation of Pruned BPE, a Byte Pair Encoding tokenizer that improves visible vocabulary efficiency by pruning low-exposure tokens while preserving internal merge structure.
tokenizer text-processing tokenization bpe byte-pair-encoding subword-tokenization pre-tokenization pruned-bpe vocabulary-pruning
-
Updated
Jul 12, 2026 - Python