pg_tokenizer

pg_tokenizer is a standalone multilingual text analyzer and vocabulary tokenizer extension for PostgreSQL, designed to power full-text BM25 ranking with vchord_bm25 and hybrid retrieval with vchord.

Website | Tokenizer Docs | BM25 Docs | vchord | vchord_bm25 | PGEXT | Pigsty (中文) | Releases | Issues

---

Vchord Suite

pg_tokenizer is part of the Vchord extension family maintained by PGSTY:

• vchord (pgsty/vchord · PGEXT · Pigsty): Scalable, high-performance, and disk-friendly vector similarity search (vchordrq & vchordg).
• vchord_bm25 (pgsty/vchord_bm25 · PGEXT · Pigsty): Native BM25 full-text ranking index (Block-WeakAnd).
• pg_tokenizer (pgsty/pg_tokenizer · PGEXT · Pigsty): Standalone multilingual text analyzer and vocabulary tokenizer.

---

Features

• Modular Text Analysis Pipeline: Chain character filters (to_lowercase, unicode_normalization), pre-tokenizers (unicode_segmentation, regex, Chinese jieba), and token filters (stopwords, Snowball stemmer, synonym, pg_dict, ngram) via declarative TOML configurations.
• Built-in & Hugging Face Vocabularies: Ship with built-in pre-trained models (bert_base_uncased, wiki_tocken, gemma2b, llmlingua2) or import any Hugging Face tokenizer.json directly into PostgreSQL.
• Corpus-Trained Custom Models: Automatically build domain-specific vocabularies from your own table columns and maintain token embeddings via database triggers (create_custom_model_tokenizer_and_trigger).
• Multilingual Support: Native support for English, European languages (30 Snowball stemmers), Chinese (jieba), Japanese/Korean (lindera), and multilingual LLM vocabularies.

---

Next version: 0.1.2 (unreleased)

The source tree adds transaction-serialized custom vocabulary writes, database-owner catalog privileges, optional corpus filters, and native PostgreSQL tsvector output. Existing tokenize() calls still return INT[]. See custom models, native full-text search, and upgrade instructions.

Quick Start

For full setup instructions (Debian/ZIP packages and building from source), see the Installation Guide or docs/01-installation.md.

1. Install via Native Linux Packages (pig)

On EL 8/9/10, Debian 12/13, and Ubuntu 22.04/24.04 (PostgreSQL 14–18), install pre-built native packages from the Pigsty repository (see package matrix on pgext.cloud):

curl -fsSL https://repo.pigsty.io/pig | bash
pig repo add pgsql -u
pig ext add pg_tokenizer

Then add pg_tokenizer.so to shared_preload_libraries and restart PostgreSQL:

psql -U postgres -c 'ALTER SYSTEM SET shared_preload_libraries = "pg_tokenizer.so"'
sudo systemctl restart postgresql

2. Run with Docker

You can also start a ready-to-use container with the tensorchord/vchord-suite image, which bundles PostgreSQL, vchord, vchord_bm25, and pg_tokenizer:

docker run \
  --name vchord-suite \
  -e POSTGRES_PASSWORD=postgres \
  -p 5432:5432 \
  -d tensorchord/vchord-suite:pg18-latest
  # Or from GHCR: ghcr.io/tensorchord/vchord-suite:pg18-latest

Connect to the database using psql:

psql -h localhost -p 5432 -U postgres

3. Basic SQL Usage

Enable the extension and add tokenizer_catalog to search_path:

CREATE EXTENSION IF NOT EXISTS pg_tokenizer;
SET search_path TO "$user", public, tokenizer_catalog;

Create a tokenizer backed by the built-in llmlingua2 model and tokenize text into an integer array of token IDs:

SELECT create_tokenizer('tokenizer1', $$
model = "llmlingua2"
$$);

SELECT tokenize(
    'PostgreSQL is a powerful, open-source object-relational database system. It has over 15 years of active development.',
    'tokenizer1'
);

More SQL examples (custom corpus models, Chinese jieba, Japanese lindera) can be found in docs/03-examples.md.

When upgrading from Lindera 0.42 to 6.2, regenerate stored Lindera token arrays and rebuild their search indexes: the embedded IPADIC dictionary now assigns different token IDs. Existing model configurations remain accepted; see the Lindera upgrade notes.

---

Documentation

Full online documentation is available at vchord.pgsty.com/bm25/tokenizer, alongside the local docs in docs/:

• Tokenizer Overview & Usage (docs/04-usage.md)
• Models (Built-in, Custom, Hugging Face, Lindera) (docs/06-model.md)
• Text Analyzer Pipeline (docs/05-text-analyzer.md)
• Multilingual Segmentation & Examples (docs/03-examples.md)
• SQL & Configuration Reference (docs/00-reference.md)
• Installation · Development · Limitations

---

License

This software is licensed under the Apache License 2.0 (Apache-2.0).

• Copyright (c) 2025-2026 TensorChord Inc.
• Copyright (c) 2026 PGSTY PTE. LTD.
• Maintained by Ruohang Feng @ PGSTY